DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 5, 7, 10-14, 16, 18, 21 and 22 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by De Bruijn et al. (published as US 20220053282 A1; hereafter De Bruijn).
Regarding claim 1, De Bruijn discloses a device configured to process an audio bitstream ([0155], [0165], [0166], e.g.), the device comprising:
a memory (inherently included to store received bitstream to be further processed by a processor as discussed below, [0173]; furthermore, it is typical that extended reality application is being executed by a computer in which a memory is inherently included, [0006]), configured to store the audio bitstream representative of an audio element in an extended reality scene ([0001], [0003], [0004] e.g.); and
processing circuitry ([0173], e.g.) coupled of the memory (inherently included to provide bitstream) and configured to:
obtain a playback dimension (the type of playback device being used inherently has a dimension; the headphones have a playback dimension of a physical space in which the playback is to occur; a group of loudspeakers have a playback dimension of a physical space in which the playback is to occur; also considering the aspect ratio of the display, “second aspect ratio” in [0020], [0051], [0052], e.g., display size in [0168], display distance in [0170]) of a physical space in which playback of the audio bitstream is to occur (when experiencing the content in extended reality);
obtain a source dimension ([0010], “first aspect ratio” in [0020], [0086]) of a source space (the space occupied by the capturing device; [0010]) in which the extended reality scene is one or more of captured (XR scene is captured by the capturing device with a source dimension occupied in a source space), generated or intended for playback;
modify (audio remapper 207), based on the playback dimension and the source dimension ([0113], [0115], [0116]), a location of the audio element to obtain a modified location for the audio element ([0027], [0030], [0032], [0034], [0055], e.g.);
render (209), based on the modified location for the audio element, the audio element to one or more speaker feeds (signals for speakers, [0068], [0099]-[0101], e.g.,); and
output the one or more speaker feeds ([0068], e.g.).
Regarding claim 2, De Bruijn shows that the processing circuitry is, when configured to modify the location of the audio element, configured to: determine, based on the playback dimension and the source dimension, a rescale factor (reads on “relationship between the original and modified aspect ratio” in [0137], e.g.); and apply the rescale factor to the location of the audio element to obtain the modified location for the audio element (“apply a linear remapping of the spatial position … according to the relationship…” in [0137], e.g.,).
Regarding claim 3, De Bruijn shows that the processing circuitry is further configured to obtain, from the encoded audio data, a syntax element (“metadata which links audio elements with image objects” in [0135], [0155], [0164]-[0166], e.g.) indicating that auto rescale (e.g., yes or no for rescaling following instructions of algorithm automatically later) is to be performed for the audio element (e.g., object signal could be rescale depending on rescaling of image), and wherein the processing circuitry is, when configured to apply the rescale factor, automatically apply, for a duration in which the audio element is present for playback (reads on the duration of the audio corresponding to the object being rendered), the rescale factor to the location of the audio element to obtain the modified location for the audio element.
Regarding claim 5, De Bruijn shows the playback dimension including a width of the physical space ([0029], [0168]).
Regarding claim 7, De Bruijn teaches the processing circuitry is, when configured to render the audio element, configured to perform, based on the playback dimension and the source dimension, audio warping with respect to the audio element to preserve an angle of incidence for the audio element (direction of arrival in [0035] and the location of the video object matched the location of the spatial audio in [0117], [0134]).
Regarding claim 10, De Bruijn shows a syntax element (metadata for plural audio elements, and how the metadata determines the subsequent rescaling, [0135], [0153], [0155], [0166]).
Regarding claim 11, De Bruijn shows speakers ([0099], [0101], e.g.).
Claims 12-14, 16, 18, 21 and 22 correspond to claims 1-3, 5, 7, 10 and 11 discussed above.
Claims 23-30 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Herre et al. (published as US 20200145776 A1; hereafter Herre).
Regarding claim 23, Herre discloses a device configured to encode an audio bitstream (Fig. 3a, [0123], [0044], e.g.), the device comprising:
a memory (inherently included to store audio signals, [0033]) configured to store an audio element (reads on audio data captured by microphone, similar to 520 in Fig. 6), and
processing circuitry ([0229], e.g.) coupled to the memory, and configured to:
specify, in the audio bitstream, a syntax element (description, such as 350) indicative of a rescaling factor for the audio element, the rescaling factor indicating how a location of the audio element is to be rescaled relative to other audio elements (Fig. 4c); and
output the audio bitstream (as shown in Fig. 3a to be decoded/rendered).
Regarding claim 24, Herre shows that the audio element is defined using higher order ambisonic coefficients (HOA) that conform to a scene-based audio format (inherent property of ambisonics coefficients), and wherein the higher order ambisonic coefficients include coefficients associated with a zero order basis function (inherently included in HOA, representing the omnidirectional component of the sound field; [0037], [0045], [0054], [0115], e.g.).
Regarding claim 25, Herre shows that the processing circuitry is further configured to specify, in the audio bitstream, a syntax element indicating that auto rescale is to be performed for the audio element (e.g., this claimed limitation reads on the metadata for objects and DoAs; at the decoder/transcoder, auto rescale would automatically perform based on the detected distance, see also the contrast with metadata for ambience/diffuse/residual, [0142], Fig. 4c, [0129]-[0130]).
Regarding claim 26, Herre shows that the syntax element (description such as 350) indicating that auto rescale is to be performed for the audio element indicates that auto rescale is to be performed, for a duration in which the audio element is present for playback (reads on the duration of rendering an object, such as 1 in Fig. 4c), the rescale factor to the location of the audio element to obtain the modified location for the audio element (e.g., object 1 is at layer 1 at a position relative the reference in the middle of layer 1, but 2 layers away at a position from a new listener position as a new reference as shown in Fig. 4c).
Claims 27-30 correspond to claims 23-26 discussed before.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 4, 6, 8, 15, 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over De Bruijn.
Regarding claims 4 and 15, De Bruijn fails to explicitly show “to determine the rescale factor as the playback dimension divided by the source dimension”. However, De Bruijn clearly teaches “linear remapping” is based on the relationship between the original and modified aspect ratio ([0137]). Furthermore, an example is provided to demonstrate how an original position, (x,y) is being remapped to (ax,by). One skilled in the art would have recognized that a and b are multiplication factors indicating the rescale factors respectively as ax/x=a, and by/b=b. Thus, it would have been obvious to one of ordinary skill in the art to modify De Bruijn by dividing the new position by the original position in order to determine the rescale factor for the new audio position.
Regarding claims 6 and 17, De Bruijn fails to show a source dimension including one or more of a width, length and height. De Bruijn teaches the definition of a source aspect ratio based on a width and a height of a source ([0029]) and the source aspect ratio is being given ([0086]). One skilled in the art would have expected that instead of directly giving a source aspect ratio, the given information could be width and height of the source, with the corresponding aspect ratio being calculated without undue experimentation. Thus, it would have been obvious to one of ordinary skill in the art to modify De Bruijn by providing the source width and height instead of corresponding aspect ratio because it is considered as an engineering design preference.
Regarding claims 8 and 19, De Bruijn shows wherein the playback dimension includes a width of the physical space and a length of the physical space ([0029]), wherein the playback dimension includes a width of the source space and a length of the source space ([0029]), and wherein the processing circuitry is, when configured to render the audio element, configured to:
perform, audio warping with respect to the audio element to preserve an angle of incidence for the audio element.
De Bruijn fails to explicitly determining a playback aspect ratio and a source aspect ratio based on the respective width and length. However, De Bruijn teaches that source aspect ratio is being remapped and change to the playback aspect ratio, and also teaches that the definition of source aspect ratio and playback aspect ratio ([0029]). It is within the level of ordinary skill to determine source aspect ratio and playback aspect ratio based on the definition without undue experimentation. Thus, it would have been obvious to one of ordinary skill in the art to modify De Bruijn by determining the corresponding aspect ratio when the dimensions of the playback device and the source space are provided in order to remap the video object and the correspond audio element based on those two aspect ratio.
De Bruijn also fails to explicitly show a threshold difference between the aspect ratios. De Bruijn teaches several criteria to determine whether to perform audio warping, such as foreground vs background, salient object vs non-salient object ([0088]), smaller display vs large display [0168]). One skilled in the art would have recognized that the audio warping should be done selectively in order to provide a more natural sound with respect to the displayed image ([0024], [0031]). De Bruijn provides a source aspect ratio being compared with various playback aspect ratios ([0086]). However, when the source aspect ratio and the playback aspect ratio are the same, it would be logical, as determined by one skilled in the art, no audio warping is required as the image position would not be changed. In this scenario, the difference between the aspect ratios is zero and no audio warping should be conducted. In view of this scenario, on the other hand, if there is a difference between the aspect ratios, audio warping should be conducted, especially with significant difference between the source aspect ratio and the playback aspect ratio. For example, the original audiovisual is captured and intended for a mobile phone, while the new playback device is a 62-inch TV. Thus, it would have been obvious to one of ordinary skill in the art to modify De Bruijn by utilizing a threshold difference for determining whether to perform audio warping in order to enhance the functionality of audio warping to only perform audio warping when there is a significant threshold difference between the aspect ratio of the source and the aspect ratio of the playback device.
One example of the claimed threshold difference is zero.
Claims 9 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over De Bruijn and Peters et al. (published as US 20170110140 A1; hereafter Peters).
Regarding claims 9 and 20, De Bruijn fails to show the coefficient is being removed. De Bruijn teaches general HOA representing the captured audio scene (0083]). General HOA includes omnidirectional coefficient which is a coefficient associated with zero order basis function, and coefficients for other directions. Peters teaches that certain coefficient, such as coefficient for zero order basis could be removed from HOA ([0067]). Such action would reduce amount of coefficients required for processing while not affecting the rendered quality significantly. Thus, it would have been obvious to one of ordinary skill to modify De Bruijn in view of Peters for this advantage.
Claims 23-30 are rejected under 35 U.S.C. 103 as being unpatentable over De Bruijn in view of Lee et al. (published as US 20210132898 A1; hereafter Lee).
Regarding claim 23, De Bruijn discloses a bitstream with a syntax element (metadata, [0135], [0155], [0164]-[0166]) indicative of a rescaling factor (e.g., yes or no for rescaling later) for the audio element, the rescaling factor indicating how a location of the audio element (foreground) is to be rescaled relative to other audio elements (background, or other elements in the foreground, e.g.) and the bitstream is being outputted (to be received by the renderer, [0155]). De Bruijn further teaches that audio is being rendered based HOA components. De Bruijn fails to explicitly show a device configured to encode the audio bitstream to be outputted. In view of teaching by De Bruijn as a whole, one skilled in the art would have recognized that the bitstream is inherently a stream with encoded data. Lee is cited here to show an encoder ([Fig. 14) for encoding HOA components ([0117], [0119], [0145], [0180]) and metadata explicitly. Thus, it would have been obvious to one of ordinary skill in the art to modify De Bruijn by incorporating an encoder, as such the one taught in Lee, in order to encoding a bitstream with HOA components and metadata to be transmitted to the renderer.
Regarding claim 24, both De Bruijn and Lee fail to explicitly show coefficient associated with a zero order basis function. However, both De Bruijn and Lee teach general HOA representing the captured audio scene (0083] in De Bruijn; [0117] in Lee). General HOA includes omnidirectional coefficient which is a coefficient associated with zero order basis function, and coefficients for other directions. Thus, it would have been obvious to one of ordinary skill to modify De Bruijn and Lee by including omni-directional coefficient in order to representing the sound scene with as much as information as allowed.
Regarding claims 25 and 26, the claimed syntax element indicating that auto rescale is to be performed reads on the metadata for audio element that allows auto rescale (such as audio element being labelled as foreground, or audio element links to image to be rescaled). The claimed duration reads on the duration when the audio element is being rendered.
Claims 27-30 correspond to claims 23-26 discussed above.
Response to Arguments
Applicant's arguments filed 9/9/2025 have been fully considered but they are not persuasive.
On p. 11, applicant argued that paragraph [0009] of De Bruijn fails to disclose that the remapping audio data based on a physical space. The office disagree. Paragraph [0009] discloses many physical space for rendering video and audio, such as TV, tablets, phones with built-in speakers or VR display and headset ([0009], [0066], e.g.). Each of them occupies a physical space in which playback of the audio bitstream is to occur. On paragraph [0020], the second aspect ratio is related to the physical space of a playback device (such as a display) and the first aspect ratio is related to the source dimension of a source space in which the XR scene is captured. An object would be rendered by the display visually, which is a playback device occupied a physical space ([0052]). While rendering the video of the object, the corresponding audio would also be rendered by the built-in speakers or headphones based on the audio bitstream ([0031]-[0032], [0035], e.g.). In response to applicant’s argument for paragraph [0052], when an audio bitstream is rendered by VR headset for example, the playback dimension of a physical space in which playback of an audio bitstream is to occur reads on the playback dimension of the VR headset ([0104]). If rendered by loudspeakers ([0099]-[0103]), the claimed playback dimension reads on the dimension occupied by loudspeakers.
On p. 12, applicant argued that De Bruijn does not disclose “modifying, based on the playback dimension and the source dimension, a location of the audio element to obtain a modified location for the audio element”. The office disagrees. De Bruijn clearly states that there is a discrepancy between the video image and the correspond audio after the aspect ratio of the video image has been changed in prior art ([0024]). The system and method as disclosed in De Bruijn would overcome such defect ([0008]). De Bruijn discloses that after converting the original aspect ratio to a new aspect ratio, the audio rendering would match the location of visual object based on the new aspect ratio. Difference playback device, having different playback dimensions of physical space, would need different rendering algorithm to account for the placement of sound generation (by headphones or loudspeakers) relative to user’s head. See the paragraphs cited in the paragraph immediately above. Paragraphs [0113]-[0116], as argued by applicant, discloses that the new position of specific image object is provided for audio rendering which would provide perceived sound position matching the new position of the specific image object depending on the playback device. Playback device, with a specific dimension, would affect how the generated sound would perceive by the user. For example, the algorithm for binaural rendering by headphones (e.g., [0068]) would be different from rendering by full-size loudspeakers ([0101] discloses 5 loudspeakers as an example; [0100] discloses many algorithms available for loudspeak8er rendering). One skilled in the art would have expected that utilizing algorithm intended for loudspeakers rendering for headphones would not generate intended sound perceived location matching a specific visual object position.
In response to applicant's argument on p. 13 that De Bruijn fails to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., space in the claim means it is the space, such as concerts venues, stadiums, conference halls, … etc, that an XR scene is captured) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
On p. 13, applicant argued that De Bruijn fails to show syntax element indicating that auto rescale is to be perform. The office disagrees. De Bruijn clearly discloses, in several paragraphs of the document, that foreground image would be treated differently from the background image when changing the original aspect ratio to a new aspect ratio ([0048], [0088], [0153]). Paragraph [0135] of De Bruijn discloses metadata indicative that an audio element is linked to a specific image object. A salient object, such as foreground, is identified by metadata. For non-salient object (such as background), another metadata would be provided identifying the non-salient object from the salient object. See paragraph [0155]). For example, the background features would be auto scaled ([0145]). See also paragraph [0149] that discloses for audio source, being inherently identified by a syntax, as being not associated with specific, narrow, and well defined position, an auto scale would be performed.
On p. 14, applicant argued that Herre does not disclose “specifying, in an audio bitstream, a syntax element …” because description is not “rescaling factor”. The office disagrees. Description in an audio bitstream includes many information about a sound scene. The most common description, aka, basic metadata is gain and location of an audio element. Herre discloses an enhanced description including a syntax element (350) indicative of a rescaling factor for an audio element relative to other audio elements. See Fig. 3b, paragraphs [0117]-[0123] and [0125]-[0129]. For example, for source located at layer 1, a rescaling factor is provided in the metadata in Fig. 4C. The metadata is indicative of a rescaling factor for the source located at layer 1 to be rescaled relative to other audio elements.
On p. 15, applicant argued that Herre does not indicate how the “scaling factor” is determined. It is irrelevant how the “scaling factor” is determined. As long as Herre discloses a scaling factor indicative of a scaling factor for an object at a position relative to other objects, Herre discloses the claimed feature.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PING LEE whose telephone number is (571)272-7522. The examiner can normally be reached Monday-Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vivian Chin can be reached at 571-272-7848. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PING LEE/ Primary Examiner, Art Unit 2695
