DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description:
Reference “801” in Fig. 8.
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 4-10, and 13-19 are rejected under 35 U.S.C. 102(a1)/102(a2) as being anticipated by Wang et al. (US 20190306451 A1, hereinafter “Wang”).
Regarding claim 1,
Wang teaches:
A scene synthesis system: (Wang: Fig.2 ref 102: spatial audio system). Wang’s spatial audio system augments the visual content using the spatial audio signal generated by ambisonic synthesizer engine 124. Wang’s ambisonics synthesizer engine includes the weight prediction layer 308 (which is the cross-model bridge), and the model layer 310 (audio neural network) to that is responsible in generating the spatial audio using information generated by the weight prediction layer. (Wang: ¶25, "In some examples, the spatial audio system generates visual and audio representations as described above and the visual or audio representations can indicate a position or location of a sound source in the content. . ."; Wang: ¶86, “In block 410, the visual content is augmented using the spatial audio signal. In some examples, the spatial audio generation application 140 causes the spatial audio system 102 to augment the visual content using the spatial audio signal”),
comprising:
a visual neural network (Wang: Fig. 2 ref 120: Video Encoder Engine 120). Wang teaches a Video Encoder Engine 120 that uses a neural network (Wang: ¶59, "In some examples, in block 402, the video encoder engine 120 can extract or generate a visual representation of the visual content or a frame of the visual content by using one or more predictive models (e.g., a neural network, deep learning model, etc.) that can be used to encode the visual content or the frame of the visual content into a visual representation that indicates or describes elements in the visual content or frame. . .");
a cross-model bridge (Wang: Fig. 4 ref 308: Weight Prediction Layer 308), Wang discloses a weight prediction layer 308 that acts as a “cross-model bridge” that process information collected by the video encoder engine (visual neural network). The weight prediction layer determine or predict a weight or bias information to be processed by the model layer 310 which is the audio neural network. Note that the weight prediction layer 308 (cross-model bridge) and the model layer 310 (audio neural network) are part of Wang’s ambisonic synthesizer engine 124. (Wang: ¶79, "The ambisonic synthesizer engine 124 can also include a weight prediction layer 308 that can be configured to determine or predict a weight W(t) or bias b(t) to be applied to a stereo audio signal based on a location or position of a source of a sound associated with the stereo audio signal as described above. In some examples, the weight W(t) or bias b(t) to be applied to a stereo audio signal can represent any algorithm, function, value, etc. that can be applied to a stereo audio signal. For instance, the ambisonic synthesizer 124 can determine or predict the weight W(t) or bias b(t) to be applied to an audio element (e.g., an individual stereo audio signal) based on a location or position of a source of a sound associated with the audio element that is indicated by the visual or audio representation in block 710 of FIG. 7"; note: the cross-model bridge is the weight prediction layer which is a part of the ambisonic synthesizer engine);
and an audio neural network (Wang: Fig. 4 ref 310: model layer 310) Wang discloses a model layer 310, which is a component of the ambisonics synthesizer engine, that processes parameters (weight, bias) generated by the weight prediction layer 308 and generates the spatial audio. In Wang: ¶19, Wang discloses that various aspects and features of their invention relate to using a predictive model such as a neural network to generate spatial audio. Since the model layer 310 is the component that generates the spatial audio, and can include three stacked convolutional layers (fundamental to convolutional neural network) as described in Wang: ¶84, the model layer 310 is using a neural network, and therefore is the audio neural network (Wang: ¶82, “For example, and with reference to FIG. 4, the ambisonic synthesizer engine 124 can include a model layer 310 that receives as an input the weight W(t) or bias b(t) to be applied to a stereo audio signal from the weight prediction layer 308, along with the stereo signal I(t) output from the stereo augmentation layer 306. The model layer 310 can apply one or more algorithms such as, for example, a linear model (neural network), to the stereo signal I(t) and a corresponding weight W(t) or bias b(t) for the stereo signal to generate a spatial audio signal”; Wang: ¶84, “In some examples, the model layer 310 can include three stacked convolutional layers each with a kernel size of 1×1. . .”; Wang: ¶19, "Various aspects and features of the present disclosure relate to using a predictive model (e.g., a neural network, deep learning model, etc.) to generate spatial (e.g., three-dimensional) audio from non-spatial audio"; Wang: ¶53, ". . .the ambisonic synthesizer engine 124 are part of a single engine or predictive model. . .");
wherein parameters of the audio neural network are generated by the cross-model bridge based on analysis of a three-dimensional visual environment modeled by the visual neural network. The weight prediction layer 308 (cross-model bridge) generates parameters (Wang: ¶79), which are then used by the model layer 310 (audio neural) network to generate spatial audio (Wang: ¶82-84;). Fig. 4 of Wang illustrates that the weight prediction layer 308 (cross-model bridge) uses content features 302 and stereo audio data 304 associated with the content as input to generate the parameters. The content features 302 are visual representations or audio representations generated by the (visual neural network) video encoder engine 120. The video encoder engine 120 analyzes content obtained (content data 114) data and extracts or generates one or more visual representations (modeled by the visual neural network) based on the content. Content data 114 include visual content such as obtained from an augmented or virtual reality environments which supports 3D environment (Wang: ¶39-40). Also, as described in Wang: ¶21, the visual content obtained from a user device can include a “three-dimensional video. Therefore, the weight prediction layer 308 (cross-model bridge) generates parameters based on analysis of a three-dimensional visual environment modeled by the visual neural network (Wang: ¶21, “In one illustrative example, a user can use a user device (e.g., a mobile device that includes a 360 degree camera) to capture visual content, along with mono audio of the visual content. The visual content can include, for example, a three-dimensional video. In this example, a spatial audio system executed on the user device can use a predictive model to analyze the visual content and the mono audio and generate spatial audio signals using the mono audio. Continuing with this example, the user can then upload the visual content and the ambisonic audio signals to a content data network, such as, YouTube”; Wang: ¶39, “For example, the spatial audio generation application 140 causes the spatial audio system 102 to obtain (e.g., receive) content data 114 indicating content to be provided to a user of the computing device 104. In some examples, the spatial audio generation application 140 causes the spatial audio system to obtain the content data 114 from the data storage unit 112, the user device 110, via user input (e.g., if a user programs the computing device 104 to include the content data 114), or any other source. The content data 114 can include content that can be provided to a user such as, for example, visual content (e.g., a virtual or augmented reality environment). In some examples, the spatial audio generation application 140 causes the spatial audio system 102 to obtain or receive audio data associated with the content from the data storage unit 112, the user device, via user input (e.g., if a user programs the computing device 104 to include the audio data), or any other source. The audio data or file associated with the content can indicate one or more stereo audio signals or sounds in the content”; Wang: ¶40, “In some embodiments, the video encoder engine 120 analyzes the content obtained by the spatial audio system 102 and extracts or generates one or more visual representations based on the content. In some examples, a visual representation of the content describes, indicates, or otherwise represents visual elements of the content. For example, the video encoder engine 120 receives content data 114 indicating various frames of a virtual reality environment. The video encoder engine 120 analyzes one or more of the frames and generates a visual representation of the frame that describes, indicates, or otherwise represents elements in the frame. As an example, a frame of the virtual reality environment includes a virtual character and the video encoder engine 120 analyzes the frame and generates a visual representation of the frame that indicates that the frame includes the virtual character. In some examples, a visual representation can indicate a position or location of an element in the content (modeled by the visual neural network). As an example, the video encoder engine 120 can analyze the frame of the virtual reality environment that includes the virtual character and generate a visual representation that indicates that the frame includes the virtual character and the location or position of the virtual character within the frame or the virtual reality environment. In some examples, the video encoder engine 120 can extract or generate a video feature representation by using one or more predictive models that can be used to encode a frame of the content into a visual representation that describes, indicates, or otherwise represents elements of the content”; Wang: ¶79, "The ambisonic synthesizer engine 124 can also include a weight prediction layer 308 that can be configured to determine or predict a weight W(t) or bias b(t) to be applied to a stereo audio signal based on a location or position of a source of a sound associated with the stereo audio signal as described above. In some examples, the weight W(t) or bias b(t) to be applied to a stereo audio signal can represent any algorithm, function, value, etc. that can be applied to a stereo audio signal. For instance, the ambisonic synthesizer 124 can determine or predict the weight W(t) or bias b(t) to be applied to an audio element (e.g., an individual stereo audio signal) based on a location or position of a source of a sound associated with the audio element that is indicated by the visual or audio representation in block 710 of FIG. 7"; Wang: ¶80, “Returning to FIG. 5, in block 408, a spatial audio signal is generated based on the weight W(t) or bias b(t) (e.g., the weight or bias determined in block 406) and the audio element (e.g., the audio element of block 404). In some examples, the spatial audio generation application 140 causes the ambisonic synthesizer engine 124 to generate the spatial audio signal based on the weight or bias and a stereo audio signal. In some embodiments, the ambisonic synthesizer engine 124 includes one or more instructions stored on a computer-readable storage medium and executable by processors of the computing device 104. When executed by the one or more processors, the computer-executable instructions cause the ambisonic synthesizer engine 124 to generate the spatial audio signal based on the weight and the stereo audio signal”; note: the cross-model bridge is the weight prediction layer which is a part of the ambisonic synthesizer engine which is the audio neural network. The weight prediction layer predicts a weight W(t) and bias b(t) (generated parameters) which are used by the ambisonic synthesizer engine to generate the spatial audio as described in Wang: ¶80).
Regarding claim 4, depending on 1,
Wang teaches:
The scene synthesis system of claim 1, wherein the visual neural network receives an input camera trajectory and generates a sequence of visual frames to model the three- dimensional visual environment. Wang’s system explicitly discloses that it can process “three-dimensional videos and generates visual representation of the frame (note: what the camera sees at a single point in time) that indicates “position or location” (trajectory) of a visual element in the content, (Wang: ¶23). When the camera is moved, then the camera trajectory changes as well, then the position and location of the visual elements changes as well, in relation to the representation and analysis of the frame. In order for a system to determine a three-dimensional “position or location” of a visual element within a frame, it must consider the camera trajectory, as for every time the camera moves, the visual elements in a frame will also change its position or location which is calculated by Wang’s video encoder engine (visual neural network). Wang: ¶19 describes that the generated ambisonics signals can be associated with visual content and indicate a location, depth, or position of a sound source, and also describes various sounds in all viewing directions (camera trajectory for each frame analyzed). Therefore, the camera trajectory is an inherent input in order to calculate the position or location of the visual elements. The video encoder receives content data 114 indicating various frames of a virtual reality environment, and analyzes one or more frames to generate a visual representation (three-dimensional model of the visual environment) of the frame that describes, indicates, or otherwise represents elements in the frame (Wang: ¶40, “In some embodiments, the video encoder engine 120 analyzes the content obtained by the spatial audio system 102 and extracts or generates one or more visual representations based on the content. In some examples, a visual representation of the content describes, indicates, or otherwise represents visual elements of the content. For example, the video encoder engine 120 receives content data 114 indicating various frames of a virtual reality environment (note: sequence of visual frames). The video encoder engine 120 analyzes one or more of the frames and generates a visual representation of the frame that describes, indicates, or otherwise represents elements in the frame (three-dimensional model). As an example, a frame of the virtual reality environment includes a virtual character and the video encoder engine 120 analyzes the frame and generates a visual representation of the frame that indicates that the frame includes the virtual character. In some examples, a visual representation can indicate a position or location of an element in the content. As an example, the video encoder engine 120 can analyze the frame of the virtual reality environment that includes the virtual character and generate a visual representation that indicates that the frame includes the virtual character and the location or position of the virtual character within the frame or the virtual reality environment. In some examples, the video encoder engine 120 can extract or generate a video feature representation by using one or more predictive models that can be used to encode a frame of the content into a visual representation that describes, indicates, or otherwise represents elements of the content.”).
Regarding claim 5, depending on 1,
Wang teaches:
The scene synthesis system of claim 1, wherein the visual neural network generates geometric information that is input to the cross-model bridge. As discussed in claim 1 rejection, Wang discloses a weight prediction layer 308 (cross-model bridge) that processes one or more content features 302 (input to the cross-model bridge; also see Wang: Fig. 4 which illustrates content features 302 as input to the weight prediction layer which is the cross-model bridge) which are visual representations generated by the video encoder engine (visual neural network). The video encoder engine analyzes 3d content and generates position or location of a virtual element. The position or location of a virtual element are the geometric information which is one of virtual representation generated by the video encoder engine (visual neural network) which is then used as input to the weight prediction layer (cross-model bridge) which is illustrated in Fig. 4. The weight prediction layer 308 (cross-model bridge) generates parameters based on a location or a position of a source of a sound (receives geometric information generated by the visual neural network). (Wang: ¶40, “. . .The video encoder engine 120 analyzes one or more of the frames and generates a visual representation of the frame that describes, indicates, or otherwise represents elements in the frame. As an example, a frame of the virtual reality environment includes a virtual character and the video encoder engine 120 analyzes the frame and generates a visual representation of the frame that indicates that the frame includes the virtual character. In some examples, a visual representation can indicate a position or location of an element in the content. . .”; Wang: ¶79, “The ambisonic synthesizer engine 124 can also include a weight prediction layer 308 that can be configured to determine or predict a weight W(t) or bias b(t) to be applied to a stereo audio signal based on a location or position (note: geometric information input) of a source of a sound associated with the stereo audio signal as described above. . .”; also see Fig. 4).
Regarding claim 6, depending on 5,
Wang teaches:
The scene synthesis system of claim 5, wherein the visual neural network encodes the geometric information into a feature vector for acoustic-aware audio generation. Wang’s invention is directed to generating spatial audio using a predictive model. Wang also discloses that the ambisonics signals generated can be associated with visual content indicating location, depth, or position of a sound source, and can describe various sounds in all viewing directions (Wang: ¶19 as referenced above), therefore Wang’s system is an acoustic aware audio generation system. As discussed in claim 5 rejection, Wang’s system, particularly, the video engine encoder (visual neural network) generates visual representations including geometric information (location or position of the visual elements within the analyzed frame as described in Wang: ¶40 as referenced above, location or position of a sound source as described in Wang: ¶45). Wang’s video engine encoder generates (encodes analyzed content of a frame, including position or location (geometric information) of visual elements) 2048-dimensional visual representations (into a feature vector) as disclosed in Wang: ¶58. (Wang: ¶58, “For example, the video encoder engine 120 receives content data 114 indicating various frames of a three-dimensional video that can be provided to a user. The video encoder engine 120 analyzes one or more of the frames and generates a visual representation of the frame that describes, indicates, or otherwise represents elements (e.g., visual elements) in the frame. As an example, a frame of a virtual reality environment includes a character and the video encoder engine 120 analyzes the frame and generates a visual representation of the frame that indicates that the frame includes the character. In some examples, a visual representation generated by the video encoder engine 120 in block 402 can indicate a position or location of an element in the content. As an example, the video encoder engine 120 can analyze the frame of the three-dimensional video that includes the character and generate a visual representation that indicates that the frame includes the character and the location or position of the character within the frame or the three-dimensional video. In some examples, in block 402, the video encoder engine 120 can analyze the content on a frame-by-frame basis and extract or generate 2048-dimensional visual representations”; Wang: ¶45, “The ambisonic synthesizer engine 124 can determine or predict a weight (e.g., a weight value) to be applied to each stereo audio signal. In some examples, a visual representation generated by the video encoder engine 120 or an audio representation generated by the audio encoder engine 122 can indicate a position or location of a sound source in the content. For instance, the video encoder engine 120 can analyze the frame of the virtual reality environment that includes the character making a sound and the audio encoder engine 122 can analyze stereo audio signals corresponding to the frame and generate an audio representation. In this example, the visual representation or the audio representation can indicate that the frame includes the character and the location or position of the sound source (e.g., the position or location of the virtual character within the virtual reality environment)”).
Regarding claim 7, depending on 5,
Wang teaches:
The scene synthesis system of claim 5, further comprising a convolutional neural network that extracts the geometric information. As discussed above, the visual representation of a content can indicate that the frame includes the location or position (the video encoder engine generating visual representations, which is a visual neural network, extracts geometric information) of the sound source as described in Wang: ¶45 as referenced above. Also, Wang’s video encoder engine also uses a neural network (Wang: ¶59 as referenced above). Additionally, Wang discloses that the neural network is a convolutional neural network (CNN). (Wang: ¶60, “A non-limiting example of a neural network is a convolutional neural network (CNN). When used for image or frame recognition, a CNN consists of multiple layers of small neuron collections that look at small portions of the input image or frame at a time. The results of these collections are then tiled to overlap, which provides a better representation of the original image or frame. A CNN with multiple intermediary layers is referred to as a deep convolutional neural network (DCNN)”).
Regarding claim 8, depending on 1,
Wang teaches:
The scene synthesis system of claim 1, wherein the cross-model bridge comprises a neural network, configured to analyze the three-dimensional environment modeled by the visual neural network, and generate the parameters of the audio neural network.
As established in the rejection of claim 1, Wang teaches a cross-model bridge which is the weight prediction layer 308 (Wang: Fig. 4, ¶79) configured to determine or predict a weight or bias (generate the parameters of the audio neural network (Wang’s model layer 310)) to be applied to a stereo audio signal based on a location or position of a sound source (configured to analyze the three-dimensional environment modeled by the visual neural network) (note: the location or position, which is geometric information, used by the video encoder engine (visual neural network) to generate visual representation to be used as input to the weight prediction layer 308 (cross-model bridge)) as discussed above. Wang’s weight prediction layer 308 uses a predictive model (wherein the cross-model bridge comprises a neural network) to determine a weight to be applied to the audio element based a location or position of a source of a sound associated with the audio element that is indicated by the visual or audio representation (three-dimensional environment modeled by the visual neural network (Wang’s video encoder engine)) (Fig.7 , block 710). Wang also discloses that the spatial audio system’s “various aspects and features relate to using a predictive model (e.g., a neural network, deep learning model, etc.)” to generate spatial audio (Wang:¶19). This further supports that Wang’s weight prediction layer 308 (cross-model bridge) comprises a neural network since it predicts weight or bias using a predictive model as also disclosed by Wang: Fig. 7 Block 710.
Regarding claim 9, depending on 1,
Wang teaches:
The scene synthesis system of claim 1, wherein the parameters of the audio neural network generated by the cross-model bridge comprise acoustic embeddings. In light of the applicant’s specification ¶63 as originally filed, the term “acoustic embeddings” is described as “model parameters”. As already discussed above, Wang’s weight predictive layer 308 (cross-model bridge) is configured to determine or predict a weight or bias (comprise acoustic embeddings) which are then used as input to the model layer 310 (parameters of the audio neural network generated by the cross model-bridge) (Wang: ¶79 as referenced above). Wang Fig. 4 clearly illustrates that the acoustic embeddings (weight W(t), bias (t)) are outputs generated by the cross-model bridge (weight prediction layer 308) and then used as input parameters of the audio neural network (model layer 310).
Regarding method claims 10, and 13-18,
Method claims 10, and 13-18 are drawn to the methods corresponding to the operations of using same as claimed in apparatus claims 1, and 4-9 respectively. Therefore, method claims 10, and 13-19 correspond to the operations in the apparatus of claims 1, and 4-9 respectively, and are rejected for the same reasons of anticipation as used above.
Regarding CRM claim 19,
CRM claim 19 is drawn to the CRM corresponding to the operations of using same as claimed in the apparatus of claim 1. Therefore, CRM claim 19 corresponds to the operations in the apparatus of claim 1, and is rejected for the same reasons of anticipation as used above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-3, 11-12, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Stengel et al. (US 20220191638 A1, hereinafter “Stengel”).
Regarding claim 2, depending on claim 1,
Wang teaches:
The scene synthesis system of claim 1. Specifically, Wang discloses a system comprising a visual neural network (video encoder engine), an audio neural network (model layer 310), and a cross-bridge model (weight prediction layer 308) to generate spatial audio to allow a user to experience an immersive auditory sensation when viewing or interacting with the visual content (see rejection of claim 1). Additionally, the generated ambisonic audio signals can describe various sounds in all viewing directions of the visual content. Wang’s system analyzes the visual content frame by frame and generates visual representation including position of the sound source (Wang: ¶40, 45) associated with the frame being analyzed since the spatial audio signal can convey a perception or sensation of a location, depth, or position of the sound source (note: relative position of sound source to camera) in the visual content (Wang: ¶81).
However, Wang only uses the viewing direction of the frame being analyzed to locate the sound source and does not teach the analogous art Stengel teaches.
Stengel’s invention is directed to systems and methods for a visually tracked spatial audio. At least one embodiment pertains to processing resources used to determine a head pose of a user and provide audio based on head pose of user. For example, at least one embodiment pertains to processors or computing systems used to determine a head pose of a user and provide audio based on head pose of user (Stengel: ¶2)
Stengel teaches:
A coordinate transformation module (Fig. 1, ref 104: 3D Head Pose Determination 104).
that applies a transformation (Stengel’s [paragraph 70] 3D Head Pose Determination 104 transforms a head pose of a user from a 3D camera view space) (Stengel: ¶70). The 3D head pose determination 104 generates a transformation matrix that indicates a head aligned frame in a virtual 3D space that is aligned with a display (e.g., a desktop display utilized by a user). In at least one embodiment, 3D head pose determination 104 generates one or more data objects that indicate a 3D head pose of a user from images of user's head (e.g., input frames 102) to an input camera direction (Stengel: ¶70). (Stengel: ¶70, “In at least one embodiment, 3D head pose determination 104 transforms a head pose of a user from a 3D camera view space to a display space (e.g., a space of a display utilized by user) centered at a display origin using extrinsics of a camera (e.g., camera that captured input frames 102) with respect to display. In at least one embodiment, 3D head pose determination 104 generates a transformation matrix that indicates a head aligned frame in a virtual 3D space that is aligned with a display (e.g., a desktop display utilized by a user). In at least one embodiment, 3D head pose determination 104 generates one or more data objects that indicate a 3D head pose of a user from images of user's head (e.g., input frames 102). In at least one embodiment, input frames 102 are continuously generated, in which a 3D head pose determination 104 continuously determines and outputs a 3D head pose (e.g., through one or more transformation matrices or other data) of a user from input frames 102. In at least one embodiment, a 3D head pose in a virtual world space is utilized for spatialized audio generated from any suitable audio source in a 3D audio engine with a default or measured head-related transfer function (HRTF). In at least one embodiment, a head pose is utilized by a system for visually tracked spatial audio with one or more HRTFs to simulate acoustic interactions of sound in a 3D environment (e.g., natural sound wave propagation, attenuation, and/or interactions) for a user of 3D environment, in which user may receive or be provided with sound through one or more audio devices (e.g., headphones) associated with one or more computing devices that generate 3D environment”) to synthesize (examiner note: synthesize here is read as generate)(paragraph 64, location and orientation of a camera) a new camera direction (paragraph 97, if a position and/or orientation of the camera is changed).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Wang and implement Stengel’s 3D Head Pose Determination 104, which is the coordinate transformation module to track the real-time head poses of the user viewing the content augmented by Wang. The reason of doing so is to allow audio sources that are fixed in a virtual 3D environment to appear stationary to a user, while supporting a full range of motion with six degrees-of-freedom (Stengel: ¶81).
Regarding claim 3, depending on 2,
The combination of Wang and Stengel teaches: The scene synthesis system of claim 2.
Wang teaches the parameters of the audio neural network (note: the parameters (weight W(t) or bias b(t) as discussed in claim 1 rejection) of the audio neural network are generated by the cross-model bridge (Wang: weight prediction layer 308)).
However Wang fails to teach the analogous art Stengel teaches.
Stengel teaches:
wherein the audio neural network (note: Stengel teaches an audio neural network, which is the 3D Audio Modulation 106 which takes in input audio 108 and the output from the 3D Head Pose Determination 104 to generate output audio 110 (see Stengel Fig.1). Stengel discloses in ¶72 that the 3D Audio Modulation 106 is a neural network; Stengel ¶59, “FIG. 1 is a block diagram illustrating a system 100 for visually tracked spatial audio, according to at least one embodiment. In at least one embodiment, system 100 for visually tracked spatial audio includes a 3D head pose determination 104 that obtains and processes input frames 102, and a 3D audio modulation 106 that, based on outputs from 3D head pose determination 104, generates output audio 110 from input audio 108; Stengel: ¶72, “In at least one embodiment, system 100 for visually tracked spatial audio includes at least one neural network model such as a perceptron model, a radial basis network (RBN), an auto encoder (AE), Boltzmann Machine (BM), Restricted Boltzmann Machine (RBM), deep belief network (DBN), deep convolutional network (DCN), extreme learning machine (ELM), deep residual network (DRN), support vector machines (SVM), and/or variations thereof, that perform various processes (e.g., processes of 3D head pose determination 104 and/or 3D audio modulation 106)”)
utilizes the new camera direction and the parameters (see rejection of claim 1 and claim 2; note: changing the camera direction inherently change the 3d visual environment and need to be re-analyzed by the visual neural network using the parameters of the audio neural network) of the audio neural network to synthesize a multi- channel audio signal corresponding to the new camera direction (Stengel: ¶99, “. . . rendering audio at block 606 includes generating an audio signal based, at least in part, on a determined head pose (e.g., distance, translational position, and/or orientation) and an input audio signal. . . generated audio signal is a synthesized binaural audio (note: multi-channel audio signal) signal. . . generating audio at block 606 includes generating an audio signal based, at least in part, on a magnification of a determined head pose change (e.g., magnifying a degree to which audio effects such as volume changes are applied based on a user turning away from a camera). In at least one embodiment, generating audio at block 606 includes determining a relationship of head pose to an audio source in a three-dimensional virtual space, and generating audio signal based, at least in part, on relationship of head pose to audio source (note: when head pose change, the viewing direction in the virtual world change to a new camera direction, the audio is generated based on this change in camera direction)”); (Stengel Fig. 1 shows that the neural audio network of Stengel which is the 3D Audio Modulation 106 receives input from the 3D Head Pose Determination which. The 3D Head Pose Determination transforms a head pose of a user using extrinsics of a camera (Stengel: ¶70) which can change when moved creating the new camera direction (Stengel: ¶64, “. . . camera extrinsics, also referred to as extrinsic parameters, refer to characteristics and/or parameters of a camera that are external to camera and change with respect to a frame of a world environment. . .; Stengel: ¶97, “. . . before determining a head pose if a camera is newly associated with a display screen, or if a position and/or orientation of a camera is changed in relation to a display screen. . .)
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Wang and implement Stengel’s method where Wang generates the ambisonic sound (using parameters (weight and bias) generated by the (Wang’s weight prediction layer) cross-model bridge based on sound sources and used as input audio 108 as illustrated in Stengel Fig.1 and wherein the audio neural network utilizes the new camera direction and the parameters of the audio neural network to synthesize a multi- channel audio signal corresponding to the new camera direction. The reason of doing so is to allow audio sources that are fixed in a virtual 3D environment to appear stationary to a user, while supporting a full range of motion with six degrees-of-freedom (Stengel: ¶81).
Regarding method claims 11-12,
Method claims 11-12 are drawn to the methods corresponding to the operations of using same as claimed in apparatus claims 2-3 respectively. Therefore, method claims 11-12 correspond to the operations in the apparatus of claims 2-3 respectively, and are rejected for the same reasons of obviousness as used above.
Regarding CRM claim 20,
CRM claim 20 is drawn to the CRM corresponding to the operations of using same as claimed in the apparatus of claim 2. Therefore, CRM claim 20 corresponds to the operations in the apparatus of claim 2, and is rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617