DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
Claims 1-15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1 & 7, the phrase "meaning" renders the claim indefinite because it is unclear whether the limitations following the phrase are part of the claimed invention. See MPEP § 2173.05(d). Additionally, the claims recite the following limitation “the user can only hear the stem track if the surface associated with the stem track is in the direction of the user”. It is unclear what occurs when the stern track is not in the direction of the user.
Claims 2-6 & 8-15 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter due to their dependencies on claims 1 & 7.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 4, 5, 7-10 & 12-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Heinemann et al. (U.S. Pub 2014/0328505) Hein, in view of GUERRERO (U.S. Pub 2014/0115468) hereinafter Guer, in view of SONG (U.S. Pub 2017/0019735) hereinafter Song, in view of Banta et al. (U.S. Pat 9,723,223) hereinafter Banta.
As per Claim 1, Hein teaches A wearable device comprising: non-transitory memory configured to store (Fig. 9, ¶50, ¶51, ¶54 wherein computing system may take the form of a wearable computing device wherein it includes a storage subsystem which includes one or more physical devices configured to hold data and/or instructions executable by the logic subsystem)
an audio file comprising a plurality of stem tracks, with each stem track representing different audio content of the audio file, (¶31, ¶40 wherein different sounds within an audio mix may be emphasized depending upon a location at which a user is gazing on a single display showing a single screen of content to highlight sounds associated with the object displayed at that location on the screen wherein sounds associated with the identified object in an audio mix may include specific sounds in the audio mix and may be subcomponents of the audio mix, e.g., individual audio tracks, features exposed by audio signal processing, etc)
a sensor configured to measure information associated with a pose of a user of the wearable device; (Fig. 1, ¶16, ¶17 wherein based on data received from sensor system 108, positional information of user 106 may be determined and tracked in real-time include location of a user or a portion of a user, e.g., a user's head or a body posture of the user)
a display configured to present images to an eye of the user of the wearable device; (Fig. 1, Fig. 9, ¶15,¶58 wherein output subsystem 906 may be used to present a visual representation of data held by storage subsystem 904 a user 106 positioned at a central location in environment 100 and viewing content presented on display device 104)
a speaker configured to present sounds to the user of the wearable device; and (Fig. 3A-3B, ¶22 wherein user 106 is listening to sounds associated with content presented on display device 104 via headphones 114)
a processor in communication with the non-transitory memory, the sensor, the speaker, and the display, the processor programmed with executable instructions to: (Fig. 9, ¶51, ¶53 wherein computing system 900 includes a logic subsystem 902 and a storage subsystem 904, an output subsystem 906, input subsystem 908, communication subsystem 910 wherein logic subsystem may include one or more processors configured to execute software instructions)
However, Hein does not explicitly teach wherein the audio file comprises metadata specifying visualizations as animations of the stem tracks;
Guer teaches wherein the audio file comprises metadata specifying visualizations as animations of the stem tracks; (Fig. 3, ¶28, ¶31, ¶34,¶37, ¶38 wherein once a connection is established between the tablet and the software, an audio track from the digital storage medium may be presented as an iconographic image on the screen of the tablet to represent each audio track wherein tracks represented on the screen shown here as kick drum 105 and snare drum 106 icons wherein the metadata associated with each audio track may need to be modified to incorporate the changes specified by the user wherien to provide immersive visualization and manipulation of tracks the present application would provide the ability of a user to be virtually positioned as the center of a space with the various tracks of a recording positioned in the virtual space around the user. By physically or virtually rotating within the space, the user is able to see and manipulate icons for each of the tracks and create a desired audible experience based on those tracks in a more intuitive and visual fashion wherien by animating the images based on volume and aural position, images can move out of the way and only relevant audio tracks may be displayed. Also, the system and method of the present disclosure can be used to aid in the composition of a musical piece that can be copyrighted. It can also be used to create original visual animations for music or audio. )
cause to display a virtual object comprising a plurality of surfaces associated with different stem tracks of the audio file, (Fig. 1, Fig. 2, ¶34 wherein Once one or more tracks are positioned on the tablet as desired for the particular sound origination points, the tablet user may then choose to modify the nature of the sounds generated beyond the direction of origin. For example, as shown in FIG. 1, a play button icon 101 and a stop button icon 102 may appear on a screen of tablet 100 and may be used by the user to start and stop the recording from being played. When the recording is being played, the tracks represented on the screen (shown here as kick drum 105 and snare drum 106 icons) may be muted by use of a mute button icon 103 or highlighted in the recording as a solo by use of a solo button icon 104. If the track represented by an icon 105 or 106 on the screen is playing, then the characteristics of the sound may be modified by the user through the use of various hand or finger gestures or movements.)
It would have been obvious to one having ordinary skill in the art at the time the invention was made to utilize the teaching of graphical user interface for mixing audio using spatial and temporal organization of Guer with the teaching of sound field adaptation based on user tracking of Hein because Guer teaches an improved organizing method of audio tracks as images in 2D or 3D space so that many tracks can be seen together at the same time and in a visually accurate way. Such improvements might also include animating the images based on volume and aural position, so that images representing tracks for sounds coming from one direction can move out of the way in the visualization so that only relevant audio tracks will be displayed to a user based on the direction the user is facing. (¶3)
However, Hein as modified does not explicitly teach determine a change in an orientation of the virtual object in response to a user input to reorient the virtual object; update, based at least partly on the change in the orientation of the virtual object, display of the virtual object in a three-dimensional (3D) environment; and wherein, by changing the orientation of the virtual object an audio mix is created which includes sounds associated with the surfaces in the direction, of the user.
Song teaches determine a change in an orientation of the virtual object in response to a user input to reorient the virtual object; (Fig. 27, ¶142 wherein the user can change the sound direction of a sub-sound output device selected through a touch and drag operation with respect to a plurality of sub-sound outputting devices displayed on the GUI 523)
update, based at least partly on the change in the orientation of the virtual object, display of the virtual object in a three-dimensional (3D) environment; and (Fig, 27, Fig, 28, ¶3, ¶22, ¶143, ¶145 wherein the present invention is to provide a sound output device capable of providing a three-dimensional (3D) sound along a direction desired by a user wherein in case the respective sound directions of the plurality of sub-sound outputting devices are changed through the GUI 523 while the plurality of sub-sound outputting devices are coupled to each other, the controller 200 may determine the roles of the plurality of sub-sound outputting devices within the established sound channel on the basis of the changed sound directions With reference to FIG. 28, sound field effects for a plurality of sub-sound outputting devices are provided through the GUI 524 based on the respective sound directions. As described with reference to FIG. 27, in case sound directions of the plurality of sub-sound outputting devices are changed, the corresponding sound field effects can be changed accordingly)
wherein, by changing the orientation of the virtual object an audio mix is created which includes sounds associated with the surfaces in the direction, of the user. (Fig. 17, Fig. 18. ¶110, ¶111, ¶112 wherien the controller 200 of the sound output device 100 can set up the sound channel formed in the sound output device 100 as a 5.1 channel according to the disposition of the plurality of sub-sound outputting devices. In other words, the sound output device 100 automatically forms a 5.1 sound channel customized for a user sitting in the middle of the living room based on the disposition of the plurality of sub-sound outputting devices comprising the 5.1 sound channel wherien FIG. 17 shows the respective sound directions and orientations of the plurality of sub-sound outputting devices. The horizontal direction from among sound directions indicates that the corresponding sub-sound output device in an octagonal column shape is in the upright position. And the vertical direction indicates that one of the side surfaces of the corresponding sub-sound output device in the octagonal column shape is disposed in parallel to the ground plane)
It would have been obvious to one having ordinary skill in the art at the time the invention was made to utilize the teaching of sound output device of Song with the teaching of sound field adaptation based on user tracking of Hein as modified because Song teaches providing an improved graphic user interface for selecting an electronic device supposed to provide a sound signal in association with the sound output device through the different electronic device wherein the user can perform a sound-related control function of the sound output device by using a different electronic device coupled with the sound output device. (¶21, ¶24)
However, Hein as modified does not explicitly teach wherein the stem tracks associated with different surfaces are directional meaning that the user can only hear the stem track if the surface associated with the stem track is in the direction of the user;
Banta teaches wherein the stem tracks associated with different surfaces are directional meaning that the user can only hear the stem track if the surface associated with the stem track is in the direction of the user; (Col. 3 lines 57-67, Col. 5 lines 23-33 wherien every time a user watches a specific sequence of video, they hear the same audio track that is played in conjunction with the video shown on the screen wherien the audio track that is played with the 360° or wide angle video varies depending upon the direction where the user is looking in the panoramic video stream. This direction of view may be designated DoV)
It would have been obvious to one having ordinary skill in the art at the time the invention was filed to utilize the teaching of panoramic video hosting with directional audio of Banta with the teaching of sound field adaptation based on user tracking of Hein as modified because Banta teaches an improved server includes an input node to receive video streams forming a panoramic video. The server also receives audio tracks corresponding to the video streams. A module forms an audio track based upon a combination of at least two of the audio tracks and directional viewing data. The audio track may be a stereo, mixed or surround sound audio track with volume modulation based upon the directional viewing data. An output node sends the audio track to a client device. (Abstract)
As per Claim 2, the rejection of claim 1 is hereby incorporated by reference; Hein as modified further teaches wherein the sensor comprises at least one of: gyroscope, an accelerometer, an inertial measurement unit, an eye tracking camera, or a camera configured to image an environment of the user. (¶60 wherein the input subsystem may comprise or interface with selected natural user input (NUI) componentry which may include a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity; as taught by Hein)
As per Claim 4, the rejection of claim 1 is hereby incorporated by reference; Hein as modified further teaches wherein stem tracks of the plurality of stem tracks are associated with different musical instruments or a vocal track. (¶31, ¶40 wherein Sounds associated with the identified object in an audio mix may include specific sounds in the audio mix and may be subcomponents of the audio mix, e.g., individual audio tracks, features exposed by audio signal processing, etc wherein if a user is watching concert footage, a volume of drums in the mix may be increased if the user is gazing at a drummer displayed on the display; as taught by Hein)
As per Claim 5, the rejection of claim 1 is hereby incorporated by reference; Hein as modified further teaches wherein at least one stem track of the plurality of stem tracks comprises information associated with a spatial location and a spatial directionality of the stem track. (Fig. 1, ¶28, ¶30 wherein an audio track from the digital storage medium may be presented as an iconographic image on the screen of the tablet to represent each audio track wherein if the image is positioned by the user in the middle of the screen, then the track(s) represented by the image may be balanced between the left and right. If the image is moved by the user toward the left side of the screen, the software would move the balance toward the left wherein the vertical arrangement of icons may be used to designate particular effects to be applied to the track based on its relative or absolute position on the screen; as taught by Guer)
Claim 7 is similar in scope to Claim 1; therefore, Claim 7 is rejected under the same rationale as Claim 1.
Claim 8 is similar in scope to Claim 4; therefore, Claim 8 is rejected under the same rationale as Claim 4.
Claim 9 is similar in scope to Claim 5; therefore, Claim 9 is rejected under the same rationale as Claim 5.
As per Claim 10, the rejection of claim 7 is hereby incorporated by reference; Hein as modified further teaches wherein the virtual object comprises a representation of musical instruments associated with the respective stem tracks or persons playing the musical instruments associated with the respective stem tracks. (Fig. 1, Fig. 2, ¶34 wherein once one or more tracks are positioned on the tablet as desired for the particular sound origination points, the tablet user may then choose to modify the nature of the sounds generated beyond the direction of origin. For example, as shown in FIG. 1, a play button icon 101 and a stop button icon 102 may appear on a screen of tablet 100 and may be used by the user to start and stop the recording from being played. When the recording is being played, the tracks represented on the screen (shown here as kick drum 105 and snare drum 106 icons) may be muted by use of a mute button icon 103 or highlighted in the recording as a solo by use of a solo button icon 104. If the track represented by an icon 105 or 106 on the screen is playing, then the characteristics of the sound may be modified by the user through the use of various hand or finger gestures or movements.; as taught by Guer)
As per Claim 12, the rejection of claim 1 is hereby incorporated by reference; Hein as modified further teaches wherein the pose comprises at least one of: ahead pose, an eye pose, a body pose, or a hand gesture. (Fig. 1, ¶16, ¶17 wherein based on data received from sensor system 108, positional information of user 106 may be determined and tracked in real-time include location of a user or a portion of a user, e.g., a user's head or a body posture of the user, and user gestures; as taught by Hein)
As per Claim 13, the rejection of claim 7 is hereby incorporated by reference; Hein as modified further teaches comprising updating the virtual object by emphasizing or de-emphasizing at least one portion of the virtual object associated with a directional stem track. (Fig. 2, ¶35 wherein if a particular track in a recording is muted or not audible at particular points during the playback, then the icon representing the track may disappear from the screen and then reappear when the track becomes audible again. As tracks are raised or lowered in volume, the icon on screen may be altered in opacity to accurately represent the volume level at any moment in time; as taught by Guer)
As per Claim 14, the rejection of claim 7 is hereby incorporated by reference; Hein as modified further teaches further comprising: receiving a selection of a stem track of the plurality of stem tracks; (Fig. 3, ¶36 wherein if the user wished to only modify the characteristics for a subset of the visible tracks, the user may use a point gesture 110 with one hand to select the desired tracks (indicated by a visual feedback such as a circle or oval 111 about the icon(s) on the screen or some other manner of visually indicated the selected tracks) and a point gesture 110 with the other hand to make the desired changes to only those selected track; as taught by Guer)
receiving an indication to modify the stem track, wherein the indication comprises a change in the pose of the user or an input from a user input device; and modifying the stem track based at least partly on the indication. (Fig. 2, ¶35 wherein if a particular track in a recording is muted or not audible at particular points during the playback, then the icon representing the track may disappear from the screen and then reappear when the track becomes audible again. As tracks are raised or lowered in volume, the icon on screen may be altered in opacity to accurately represent the volume level at any moment in time; as taught by Guer)
As per Claim 15, the rejection of claim 14 is hereby incorporated by reference; Hein as modified further teaches where in modifying the stem track comprising at least one of: adjusting the volume, replacing the stem track with another stem track, muting the stem track, emphasizing the stem track over other stem tracks of the plurality of stem tracks. (Fig. 2, ¶35 wherein if a particular track in a recording is muted or not audible at particular points during the playback, then the icon representing the track may disappear from the screen and then reappear when the track becomes audible again. As tracks are raised or lowered in volume, the icon on screen may be altered in opacity to accurately represent the volume level at any moment in time; as taught by Guer)
Claim 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hein in view of Guer in view of Song in view of Banta as applied to claim 1 above, and further in view of Ooi (U.S. Pub 2015/0022444).
As per Claim 3, the rejection of claim 1 is hereby incorporated by reference; Hein as modified previously taught the wearable device, stem tracks and environment of the user. However, Hein as modified does not explicitly teach wherein the wearable device is configured to recognize a physical object in an environment of the user, and to present a graphical representation of a stem track of the plurality of stem tracks as being associated with the physical object.
Ooi teaches wherein the wearable device is configured to recognize a physical object in an environment of the user, and to present a graphical representation of a stem track of the plurality of stem tracks as being associated with the physical object. (Fig. 8, ¶107, ¶108, ¶109, ¶110, ¶111 wherein the HMD worn by the user superimposes an image (reference image, or reference scene) captured by the camera CA (which is included by the display device in the example illustrated in FIG. 8), and displays a virtual object V_OBJ in part of an augmented reality scene wherein The user moves the CD, which is the real object R_OBJ, to bring it close to the application window, which is the virtual object V_OBJ. At this point, the real object R_OBJ is captured by the imaging unit 101, and its position is analyzed by the three-dimensional information analysis unit 102. Further, the positional relationship analysis unit 105 detects that the real object R_OBJ and the virtual object V_OBJ have been brought close together, and outputs a predetermined signal wherein When the signal indicating that the real object R_OBJ and the virtual object V_OBJ have been brought close together is output from the positional relationship analysis unit 105, a predetermined command is issued from the command issuance unit 106 wherein a command for performing playback of a song included on the CD is issued as the predetermined command, and the song included on the CD is played from a speaker, which is the output device 210)
It would have been obvious to one having ordinary skill in the art at the time the invention was made to utilize the teaching of information processing apparatus and method of Ooi with the teaching of sound field adaptation based on user tracking of Hein as modified because Ooi teaches providing a novel and improved information processing apparatus, information processing method, and program that enable the direct operation of virtual information or a virtual object by utilizing a real object that exists in the real world wherein including processing circuitry that is configured to issue a control command relating to a real object based on a displayed positional relationship between the real object and a predetermined object. (¶6, ¶7)
Claims 6 & 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hein in view of Guer in view of Song in view of Banta as applied to claims 1 & 7 above, and further in view of Bickerstaff et al. (U.S. Pub 2016/0035140) hereinafter Bick.
As per claim 6, the rejection of claim 1 is hereby incorporated by reference; Hein as modified previously taught the virtual object and the user. However, Hein as modified does not explicitly teach wherein at least some of the virtual object is presented at different depth planes relative to the user.
Bick teaches wherein at least some of the virtual object is presented at different depth planes relative to the user. (Fig. 18, ¶112, ¶113 wherein a depth map which may be derived, for example, from image data captured by a 3-D (binocular) camera or a so-called Z-camera, or which may be generated as part of the operation of a computer games machine's rendering engine wherein three image objects labelled as objects A, B and C, are shown at respective image depths measured from an arbitrary depth position 700 of z.sub.A, z.sub.B and z.sub.C)
It would have been obvious to one having ordinary skill in the art at the time the invention was made to utilize the teaching of image processing of Bick with the teaching of sound field adaptation based on user tracking of Hein as modified because Bick teaches an immersive video display such as a head-mounted display (HMD) is capable of reproducing images life-sized and in their correct orientation to the viewer. For interactive applications, the image reprojection can be performed rapidly and at the latest time just before display. This can reduce unwanted image movement caused by rendering latency. For any application, this technique can stabilize camera motion reducing viewer sickness and maintains the real world orientation of objects, improving immersion and realism. (¶126)
Claim 11 is similar in scope to Claim 6; therefore, Claim 11 is rejected under the same rationale as Claim 6.
Related Art
Related Art not relied upon Nguyen (U.S. Pub 2014/0274388) for teaching a sensor 110 may track the player's eyes gaze direction. When analyzed in the context of the player's location information from player location database 132 and the venue layout database 134, the player's gaze may be mapped to a screen in the sports bar displaying a football game. Rimon (U.S. Pub 2014/0364228) for teaching tracking the player's viewing direction during gameplay and ensure presentation of regions that the player viewed during gameplay.
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANGIE BADAWI whose telephone number is (571)270-7590. The examiner can normally be reached Monday thru Wednesday 9:00am - 5:00pm EST with Thursdays and Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Fred Ehichioya can be reached at (571) 272-4034. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANGIE BADAWI/Primary Examiner, Art Unit 2179
