DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The objections to the claims have been withdrawn in view of applicants amendments filed 05/27/2026
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-11, 13-17, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Geisner et al. (U.S. Pub. No. 20130083173) in view of Seefeldt et al. (U.S. Doc. No. 6448971), further in view of Klingler et al. (U.S. Pub. No. 20200090644).
Regarding claim 1, Geisner discloses an information processing device comprising (paragraph 3, “Technology described herein provides various embodiments for a personal audiovisual (A/V) apparatus including a near-eye, augmented reality (AR) display apparatus that can provide a virtual spectator experience of an event for a user of the apparatus.”): a communication unit configured to communicate with a real world information acquisition device including an environmental sensor and an acoustic sensor (paragraph 35, “outward facing capture devices 113, e.g. cameras, for recording digital image data such as still images, videos or both, and transmitting the visual recordings to the control circuitry 136 which may in turn send the captured image data to the companion processing module 4 which may also send the data to one or more computer systems 12 or to another personal A/V apparatus over one or more communication networks 50.”; also, paragraph 34, “The frame 115 includes a nose bridge portion 104 with a microphone 110 for recording sounds and transmitting audio data to control circuitry 136.”); a point group data management unit, including one or more processors, configured to store 3D point group data of a real space acquired from the real world information acquisition device (paragraph 72, “A 3D point cloud representing the surfaces of objects including things like walls and floors in a space can be generated based on captured image data and depth data of the user environment. A 3D mesh of the surfaces in the environment can then be generated from the point cloud.”; also, paragraph 70, “For example, if the scene mapping engine 306 receives depth images from multiple cameras, the engine 306 correlates the images to have a common coordinate system by lining up the images and uses depth data to create the volumetric description of the environment.”); a sound information analysis unit, including one or more processors, configured to: analyze sound information data of ambient sound acquired from the real world information acquisition device (para 78, “Sound recognition software engine 194 of the 3D audio engine identifies audio data from the real world received via microphone 110 for application control via voice commands and for environment and object recognition. Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car.”); attribute change unit including one or more processors, configured to: virtual space construction unit, including one or more processors, configured to construct a virtual space video using the adjusted 3D point group data (para 72, “In some examples, a 3D mapping, whether it be a depth map generated by front facing cameras 113 including a display field of view, a 3D mapping of an environment or a location in a view independent coordinate system, or somewhere in between, may be modeled as a 3D mesh of an environment. A mesh may comprise a detailed geometric representation of various features including real and virtual objects and surfaces thereof within a particular environment or region of an environment.”). Geisner does not disclose based on analyzing the sound data, determine whether a type of the sound information data corresponds to an artificial sound or a natural sound, and in response to determining that the type of the sound information data corresponds to the natural sound, adjust a position of a first subset of the 3D point group data stored in the point group data management unit; or in response to determining that the type of sound information data corresponds to the artificial sound, adjust a color of a second subset of the 3D point group data stored in the point group data management unit.
However, in a similar field of endeavor, Klingler discloses based on analyzing the sound data, determine whether a type of the sound information data corresponds to an artificial sound or a natural sound (para 19, “the classifier 26 makes a binary classification and determines whether the audio signal relates to a natural sound versus an artificial sound (also referred to as a playback sound).”; also, para 29, “the machine-learning model determines a classification, e.g., whether the sound is artificial vs. natural, based on the determined features”; also, para 5, “Natural sounds include sounds such as a person speaking, a door closing, a piano playing, etc. which have been picked up directly from their "natural" source, by the microphones.”), while Seefeldt discloses in response to determining that the type of the sound information data corresponds to the natural sound, adjust a position of a first subset of the 3D point group data stored in the point group data management unit; or in response to determining that the type of sound information data corresponds to the artificial sound, adjust a color of a second subset of the 3D point group data stored in the point group data management unit (col 2, “The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal.”, also, col 1-2, “According to another aspect of the invention, the color of each vertex is changed in response to detected triggering events.”; also, col 3, “where C(s.sub.k,n) is a time-varying function which generates a color based on surface coordinates.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's invention of a personal augmented reality apparatus that generates and stores a three-dimensional point cloud of a real environment from captured image and depth data and that analyzes real-world sound using a sound recognition software engine, with the features of Klingler's invention of a microphone-signal classifier that determines whether a captured sound is a natural sound or an artificial sound, and with the features of Seefeldt's invention of audio-event-triggered color transformation of the vertices of a rendered three-dimensional surface. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Klingler teaches that "the classifier 26 makes a binary classification and determines whether the audio signal relates to a natural sound versus an artificial sound," supplying the natural-versus-artificial determination that Geisner's sound recognition software engine 194 leaves undifferentiated, so that the sound type already identified by Geisner from a microphone signal is resolved into the specific artificial-or-natural category the claim requires. Second, Seefeldt teaches that "the color of each vertex is changed in response to detected triggering events" and that "The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal," which supplies the sound-to-attribute linkage that Geisner does not disclose, so that the color attribute of a subset of the points making up Geisner's stored three-dimensional point cloud is adjusted in response to the audio event whose type Klingler has classified as artificial. Third, because Geisner already captures ambient sound at microphone 110 and renders its point cloud into a three-dimensional mesh of the environment, applying Klingler's natural-versus-artificial classifier to that captured sound and then driving Seefeldt's audio-triggered color change of the point cloud vertices on the basis of that classification produces an audio-responsive virtual space video without altering the underlying capture-and-render pipeline Geisner already provides.
Regarding claim 2, Geisner as modified by Klingler and Seefeldt discloses the information processing device according to claim 1, wherein the point group data management unit includes: a point group data acquisition unit, including one or more processors, configured to acquire the 3D point group data from the environmental sensor (Geisner: paragraph 35, “outward facing capture devices 113, e.g. cameras, for recording digital image data such as still images, videos or both, and transmitting the visual recordings to the control circuitry 136 which may in turn send the captured image data to the companion processing module 4 which may also send the data to one or more computer systems 12 or to another personal A/V apparatus over one or more communication networks 50.”; also, paragraph 72, “A 3D point cloud representing the surfaces of objects including things like walls and floors in a space can be generated based on captured image data and depth data of the user environment.”; also, para 70, “Image and depth data from multiple perspectives can be received in real time from other 3D image capture devices 20 under control of one or more network accessible computer systems 12 or from at least one other personal A/V apparatus 8 in the environment. Depth images from multiple perspectives may be combined based on a view independent coordinate system for describing an environment (e.g. an x, y, z representation of a room, a store space, or a geofenced area) for creating the volumetric or 3D mapping.”); a point group data storage unit, including one or more processors, configured to store the acquired 3D point group data (Geisner: paragraph 66, “The scene mapping engine 306 can also use a view independent coordinate system for 3D mapping. The map can be stored in the view independent coordinate system in a storage location (e.g. 324) accessible as well by other personal A/V apparatus 8, other computer systems 12 or both, be retrieved from memory and be updated over time as one or more users enter or re-enter the environment.”; also, para 66, “The map can be stored in the view independent coordinate system in a storage location (e.g. 324) accessible as well by other personal A/V apparatus 8, other computer systems 12 or both, be retrieved from memory and be updated over time as one or more users enter or re-enter the environment.”); and a position determination unit, including one or more processors, configured to acquire an installation position of the real world information acquisition device (Geisner: paragraph 49, “Inside, or mounted to temple 102, are an ear phone 130 of a set of ear phones 130, inertial sensors 132, one or more location or proximity sensors 144, some examples of which are a GPS transceiver, an infrared (IR) transceiver, or a radio frequency transceiver for processing RFID data.”; also, paragraph 49, “The inertial sensors are for sensing position, orientation, and sudden accelerations of head mounted display device 2. From these movements, head position and orientation, and thus orientation of the display device, may also be determined”; also, para 68, “For example, location data such as GPS data from a GPS transceiver 144 on the display device 2 may identify the location of the user.”).
Regarding claim 3, Geisner as modified by Klingler and Seefeldt discloses the information processing device according to claim 1, wherein the sound information analysis unit includes: a sound information acquisition unit, including one or more processors, configured to acquire the sound information data from the acoustic sensor (Geisner: paragraph 34, “The frame 115 includes a nose bridge portion 104 with a microphone 110 for recording sounds and transmitting audio data to control circuitry 136”; also, paragraph 78, “Sound recognition software engine 194 of the 3D audio engine identifies audio data from the real world received via microphone 110 for application control via voice commands and for environment and object recognition.”) a sound information analysis rule management unit, including one or more processors, configured to manage an analysis rule of the sound information data (Geisner: paragraph 78, “Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car”; also, paragraph 78, “Additionally, pre-generated 3D maps of a location can provide an audio index of sounds of objects fixed in the location or which enter and leave the location on a regular basis, e.g. train and bus sounds.”); and an analysis unit, including one or more processors, configured to analyze the acquired sound information data on the basis of the analysis rule and calculate the type of the sound information data (Geisner: paragraph 78, “Sound recognition software engine 194 of the 3D audio engine identifies audio data from the real world received via microphone 110 for application control via voice commands and for environment and object recognition. Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car.”).
Regarding claim 5, Geisner as modified by Klingler Seefeldt discloses the information processing device according to claim 1, wherein the attribute change unit includes: point group data attribute change rule management unit, including one or more processors, configured to manage a change rule of an attribute of the 3D point group data; and a point group data attribute change unit, including one or more processors, configured to change the attribute information of the 3D point group data stored in the point group data management unit on the basis of the change rule.
However, in a similar field of endeavor, Seefeldt further discloses a point group data attribute change rule management unit, including one or more processors, configured to manage a change rule of an attribute of the 3D point group data (col 3, “for each animation control signal, the parameters governing the exact behavior of the functions M(s.sub.k,n), C(s.sub.k,n), and b(s.sub.k,n) may be randomized in order to introduce visual variety. For example, in the case of FIG. 2, the spiral velocity may be chosen randomly between some preset range of values.”; also, col 3, “where M(s.sub.k,n) is a time-varying function which maps surface coordinates to texture coordinates, and b(s.sub.k,n) is a time-varying blending function that lies between 0 and 1”); and a point group data attribute change unit, including one or more processors, configured to change the attribute information of the 3D point group data stored in the point group data management unit on the basis of the change rule (col 2, “The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal.”; also, col 1, “a method for modifying a texture mapped onto a computer generated 3D object includes the acts of remapping texture coordinates to vertices of the object at discrete times during a control interval generated in response to an external triggering event. The image is interpolated across the polygons associated with vertices. By varying the image coordinates at each vertex interesting transformations of the image are achieved without the cost of modifying each individual pixel of the transformed image.”; also, col 3, “where C(s.sub.k,n) is a time-varying function which generates a color based on surface coordinates.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's invention of a personal augmented reality apparatus that stores a three-dimensional point cloud of a real environment and identifies real-world sound, as combined with Klingler's natural-versus-artificial sound classifier and Seefeldt's audio-triggered attribute transformation, with the further features of Seefeldt's invention of organizing that attribute transformation into a managed set of parametric change rules and a separate operation that applies those rules to the rendered surface. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Seefeldt expressly teaches that "the parameters governing the exact behavior of the functions M(s.sub.k,n), C(s.sub.k,n), and b(s.sub.k,n) may be randomized in order to introduce visual variety," establishing that the change rule is a managed, configurable set of parameters that can be maintained separately from the per-frame transformation, which corresponds to the recited point group data attribute change rule management unit. Second, Seefeldt separately teaches that "The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal," which is the act of applying that managed rule to change the attribute of the point cloud, corresponding to the recited point group data attribute change unit. Third, organizing Seefeldt's audio-driven attribute transformation into a rule-management sub-unit and a rule-application sub-unit mirrors the structure Seefeldt itself describes and allows the rule parameters to be managed independently of the per-frame transformation of the point cloud attributes, so that the attribute change unit already established for claim 1 is implemented as the two recited sub-units.
Regarding claim 6, Geisner as modified by Klingler and Seefeldt discloses the information processing device according to claim 1, wherein the virtual space construction unit includes: a point group data visualization unit, including one or more processors, configured to visualize the 3D point group data in which the attribute information is changed to generate video data of a virtual space (Geisner: paragraph 72, “In some examples, a 3D mapping, whether it be a depth map generated by front facing cameras 113 including a display field of view, a 3D mapping of an environment or a location in a view independent coordinate system, or somewhere in between, may be modeled as a 3D mesh of an environment. A mesh may comprise a detailed geometric representation of various features including real and virtual objects and surfaces thereof within a particular environment or region of an environment. A 3D point cloud representing the surfaces of objects including things like walls and floors in a space can be generated based on captured image data and depth data of the user environment.”) and a point group data visualization management unit, including one or more processors, configured to manage visualization of the 3D point group data on the basis of an installation position of the real world information acquisition device (Geisner: paragraph 70, “For example, if the scene mapping engine 306 receives depth images from multiple cameras, the engine 306 correlates the images to have a common coordinate system by lining up the images and uses depth data to create the volumetric description of the environment. The scene mapping engine 306 identifies the position and tracks the movement of real and virtual objects in the volumetric space based on communications with the object recognition engine 192 of the image and audio processing engine 191 and one or more executing applications 166 generating virtual objects.”; also, paragraph 68, “location data such as GPS data from a GPS transceiver 144 on the display device 2 may identify the location of the user.”; also, para 68, “For example, location data such as GPS data from a GPS transceiver 144 on the display device 2 may identify the location of the user.”).
Regarding claim 7, Geisner discloses an information processing method executed by a processor of an information processing device including the processor and a storage, comprising (paragraph 4, “The technology provides one or more embodiments of a method for providing a virtual spectator experience of an event for viewing with a near-eye, augmented reality display of a personal A/V apparatus. An embodiment of the method comprises receiving in real time one or more positions of one or more event objects participating in the event occurring at a first location which is remote from a second location, mapping the one or more positions of the one or more event objects in the first 3D coordinate system for the first location to a second 3D coordinate system for a second location remote from the first location, determining a display field of view of a near-eye, augmented reality display of a personal A/V apparatus at the second location, and sending in real time 3D virtual data representing the one or more event objects which are within the display field of view to the personal A/V apparatus at the second location.”): communicating with a real world information acquisition device including an environmental sensor and an acoustic sensor (paragraph 35, “outward facing capture devices 113, e.g. cameras, for recording digital image data such as still images, videos or both, and transmitting the visual recordings to the control circuitry 136 which may in turn send the captured image data to the companion processing module 4 which may also send the data to one or more computer systems 12 or to another personal A/V apparatus over one or more communication networks 50.”; also, paragraph 34, “The frame 115 includes a nose bridge portion 104 with a microphone 110 for recording sounds and transmitting audio data to control circuitry 136.”); storing 3D point group data of a real space acquired from the real world information acquisition device in the storage (paragraph 72, “A 3D point cloud representing the surfaces of objects including things like walls and floors in a space can be generated based on captured image data and depth data of the user environment.”; also, paragraph 66, “The scene mapping engine 306 can also use a view independent coordinate system for 3D mapping. The map can be stored in the view independent coordinate system in a storage location (e.g. 324) accessible as well by other personal A/V apparatus 8, other computer systems 12 or both, be retrieved from memory and be updated over time as one or more users enter or re-enter the environment.”); analyzing sound information data of ambient sound acquired from the real world information acquisition device (para 78, “Sound recognition software engine 194 of the 3D audio engine identifies audio data from the real world received via microphone 110 for application control via voice commands and for environment and object recognition. Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car.”); virtual space video using the adjusted 3D point group data (para 72, “In some examples, a 3D mapping, whether it be a depth map generated by front facing cameras 113 including a display field of view, a 3D mapping of an environment or a location in a view independent coordinate system, or somewhere in between, may be modeled as a 3D mesh of an environment. A mesh may comprise a detailed geometric representation of various features including real and virtual objects and surfaces thereof within a particular environment or region of an environment.”). Geisner does not disclose based on analyzing the sound data, determining whether a type of the sound information data corresponds to an artificial sound or a natural sound; in response to determining that the type of the sound information data corresponds to the natural sound, adjusting a position of a first subset of the 3D point group data stored in the storage; or in response to the determining that the type of sound information data corresponds to the artificial sound, adjusting a color of a second subset of the 3D point group data stored in the storage.
However, in a similar field of endeavor, Klingler discloses based on analyzing the sound data, determining whether a type of the sound information data corresponds to an artificial sound or a natural sound (para 19, “the classifier 26 makes a binary classification and determines whether the audio signal relates to a natural sound versus an artificial sound (also referred to as a playback sound)”; also, para 29, “the machine-learning model determines a classification, e.g., whether the sound is artificial vs. natural, based on the determined features”); while Seefeldt discloses in response to determining that the type of the sound information data corresponds to the natural sound, adjusting a position of a first subset of the 3D point group data stored in the storage; or in response to the determining that the type of sound information data corresponds to the artificial sound, adjusting a color of a second subset of the 3D point group data stored in the storage (col 2, “The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal.”; also, col 1-2, “According to another aspect of the invention, the color of each vertex is changed in response to detected triggering events”; also, col 3, “where C(s.sub.k,n) is a time-varying function which generates a color based on surface coordinates.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's method of operating a personal augmented reality apparatus that generates and stores a three-dimensional point cloud of a real space from captured image and depth data while identifying real-world sound via a sound recognition software engine, with the features of Klingler's invention of classifying a microphone sound as a natural sound or an artificial sound, and with the features of Seefeldt's invention of changing the color of the vertices of a rendered three-dimensional surface in response to events detected in an audio signal. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Klingler teaches that "the machine-learning model determines a classification, e.g., whether the sound is artificial vs. natural, based on the determined features," which supplies the natural-versus-artificial determination that Geisner's sound recognition leaves undifferentiated. Second, Seefeldt teaches that "the color of each vertex is changed in response to detected triggering events," providing the sound-to-attribute linkage missing from Geisner's method, so that the color of a subset of the points of Geisner's stored point cloud is adjusted in response to the audio event Klingler has classified as artificial. Third, because Geisner already captures ambient sound and renders its stored point cloud into a three-dimensional mesh, applying Klingler's classifier to that sound and Seefeldt's audio-triggered color change to the point cloud is a routine application that yields an audio-responsive virtual space video using Geisner's existing capture-and-render pipeline.
Regarding claim 8, Geisner as modified by Klingler and Seefeldt discloses One or more non- transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising (Geisner: para 125, “The example computer systems illustrated in the figures include examples of computer readable storage devices. A computer readable storage device is also a processor readable storage device. Such devices may include volatile and nonvolatile, removable and non-removable memory devices implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.”): communicating with a real world information acquisition device including an environmental sensor and an acoustic sensor (Geisner: para 35, “outward facing capture devices 113, e.g. cameras, for recording digital image data such as still images, videos or both, and transmitting the visual recordings to the control circuitry 136 which may in turn send the captured image data to the companion processing module 4 which may also send the data to one or more computer systems 12 or to another personal A/V apparatus over one or more communication networks 50.”; also, para 34, “The frame 115 includes a nose bridge portion 104 with a microphone 110 for recording sounds and transmitting audio data to control circuitry 136.”); storing 3D point group data of a real space acquired from the real world information acquisition device in the storage (Geisner: para 72, “A 3D point cloud representing the surfaces of objects including things like walls and floors in a space can be generated based on captured image data and depth data of the user environment. A 3D mesh of the surfaces in the environment can then be generated from the point cloud.”; also, para 66, “The map can be stored in the view independent coordinate system in a storage location (e.g. 324) accessible as well by other personal A/V apparatus 8, other computer systems 12 or both, be retrieved from memory and be updated over time as one or more users enter or re-enter the environment.”); analyzing sound information data of ambient sound acquired from the real world information acquisition device (Geisner: para 78, “Sound recognition software engine 194 of the 3D audio engine identifies audio data from the real world received via microphone 110 for application control via voice commands and for environment and object recognition. Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car.”); virtual space video using the adjusted 3D point group data (Geisner: para 72, “In some examples, a 3D mapping, whether it be a depth map generated by front facing cameras 113 including a display field of view, a 3D mapping of an environment or a location in a view independent coordinate system, or somewhere in between, may be modeled as a 3D mesh of an environment. A mesh may comprise a detailed geometric representation of various features including real and virtual objects and surfaces thereof within a particular environment or region of an environment.”). Geisner does not disclose based on analyzing the sound data, determining whether a type of the sound information data corresponds to an artificial sound or a natural sound; in response to determining that the type of the sound information data corresponds to the natural sound, adjusting a position of a first subset of the 3D point group data stored in the storage; or in response to determining that the type of sound information data corresponds to the artificial sound, adjusting a color of a second subset of the 3D point group data stored in the storage.
However, in a similar field of endeavor, Klingler discloses based on analyzing the sound data, determining whether a type of the sound information data corresponds to an artificial sound or a natural sound (para 19, “the classifier 26 makes a binary classification and determines whether the audio signal relates to a natural sound versus an artificial sound (also referred to as a playback sound).”; also, para 45, “the natural versus artificial sound discriminator includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision”); while Seefeldt discloses in response to determining that the type of the sound information data corresponds to the natural sound, adjusting a position of a first subset of the 3D point group data stored in the storage; or in response to determining that the type of sound information data corresponds to the artificial sound, adjusting a color of a second subset of the 3D point group data stored in the storage (col 2, “The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal.”; also, col 1-2, “According another aspect of the invention, the color of each vertex is changed in response to detected triggering events”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's invention of a personal augmented reality apparatus implemented as computer readable instructions stored on computer readable storage devices that generates a three-dimensional point cloud of a real environment and identifies real-world sound, with the features of Klingler's invention of classifying a microphone sound as a natural sound or an artificial sound, and with the features of Seefeldt's invention of changing the color of the vertices of a rendered three-dimensional surface in response to events detected in an audio signal. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Klingler teaches that "the natural versus artificial sound discriminator includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision," supplying the natural-versus-artificial determination that Geisner does not perform. Second, Seefeldt teaches that "the color of each vertex is changed in response to detected triggering events," supplying the program-level operation of adjusting the color of a subset of the stored point cloud in response to the audio event Klingler has classified as artificial. Third, because Geisner's information processing device is already implemented through software executing on one or more processors, incorporating Klingler's classifier and Seefeldt's audio-triggered color-change functions as additional program logic is a straightforward extension of Geisner's existing program architecture that yields an audio-responsive virtual space video.
Regarding claim 9, Geisner as modified by Klingler and Seefeldt discloses the information processing method of claim 7, wherein storing 3D point group data of a real space acquired from the real world information acquisition device in the storage comprises: acquiring the 3D point group data from the environmental sensor (Geisner: para 70, “Image and depth data from multiple perspectives can be received in real time from other 3D image capture devices 20 under control of one or more network accessible computer systems 12 or from at least one other personal A/V apparatus 8 in the environment. Depth images from multiple perspectives may be combined based on a view independent coordinate system for describing an environment (e.g. an x, y, z representation of a room, a store space, or a geofenced area) for creating the volumetric or 3D mapping.”); storing the acquired 3D point group data (Geisner: para 66, “The map can be stored in the view independent coordinate system in a storage location (e.g. 324) accessible as well by other personal A/V apparatus 8, other computer systems 12 or both, be retrieved from memory and be updated over time as one or more users enter or re-enter the environment.”); and acquiring an installation position of the real world information acquisition device (Geisner: para 49, “Inside, or mounted to temple 102, are an ear phone 130 of a set of ear phones 130, inertial sensors 132, one or more location or proximity sensors 144, some examples of which are a GPS transceiver, an infrared (IR) transceiver, or a radio frequency transceiver for processing RFID data.”; also, para 68, “For example, location data such as GPS data from a GPS transceiver 144 on the display device 2 may identify the location of the user.”).
Regarding claim 10, Geisner as modified by Klingler and Seefeldt discloses the information processing method of claim 7, wherein analyzing sound information data of ambient sound acquired from the real world information acquisition device comprises: acquiring the sound information data from the acoustic sensor (Geisner: para 34, “The frame 115 includes a nose bridge portion 104 with a microphone 110 for recording sounds and transmitting audio data to control circuitry 136.”); and managing an analysis rule of the sound information data (Geisner: para 78, “Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car. Additionally, voice data files stored in user profile data 197 or user profiles 322 may also identify a speaker with whom a person object mapped in the environment may be associated. In addition to uploading their image data, personal A/V apparatus 8 and 3D image capture devices 20 in a location upload their captured audio data to a network accessible computing system 12. Additionally, pre-generated 3D maps of a location can provide an audio index of sounds of objects fixed in the location or which enter and leave the location on a regular basis, e.g. train and bus sounds”).
Regarding claim 11, Geisner as modified by Klingler and Seefeldt discloses the information processing method of claim 10, wherein determining whether a type of the sound information data corresponds to an artificial sound or a natural sound comprises: analyzing the acquired sound information data on the basis of the analysis rule (Geisner: para 78, “Sound recognition software engine 194 of the 3D audio engine identifies audio data from the real world received via microphone 110 for application control via voice commands and for environment and object recognition”); and calculating the type of the sound information data (Geisner: para 78, “Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car.”).
Regarding claim 13, Geisner as modified by Klingler and Seefeldt discloses the information processing method of claim 7, further comprising: change rule of an attribute of the 3D point group data; and changing the attribute of the 3D point group data in the storage on the basis of the change rule.
However, in a similar field of endeavor, Seefeldt discloses managing a change rule of an attribute of the 3D point group data (col 3, “for each animation control signal, the parameters governing the exact behavior of the functions M(s.sub.k,n), C(s.sub.k,n), and b(s.sub.k,n) may be randomized in order to introduce visual variety. For example, in the case of FIG. 2, the spiral velocity may be chosen randomly between some preset range of values.”); and changing the attribute of the 3D point group data in the storage on the basis of the change rule (col 2, “The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's method, as combined with Klingler's natural-versus-artificial sound classifier and Seefeldt's audio-triggered attribute transformation, with the further features of Seefeldt's invention of maintaining a managed set of parametric change rules and separately applying those rules to change the attribute of the rendered surface. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Seefeldt teaches that "the parameters governing the exact behavior of the functions M(s.sub.k,n), C(s.sub.k,n), and b(s.sub.k,n) may be randomized in order to introduce visual variety," establishing that the change rule is a managed, configurable set of parameters, which corresponds to the recited managing of a change rule. Second, Seefeldt teaches that "The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal," which is the act of applying that managed rule to change the attribute of the point cloud. Third, separating the management of the parametric rule from the per-frame application of that rule mirrors the structure Seefeldt itself describes and allows the rule parameters to be maintained independently of the per-frame transformation of Geisner's stored point cloud.
Regarding claim 14, Geisner as modified by Klingler and Seefeldt discloses the information processing method of claim 7, wherein constructing the virtual space video using the adjusted 3D point group data comprises: visualizing the 3D point group data in which the attribute information is modified to generate video data of a virtual space (Geisner: para 72, “In some examples, a 3D mapping, whether it be a depth map generated by front facing cameras 113 including a display field of view, a 3D mapping of an environment or a location in a view independent coordinate system, or somewhere in between, may be modeled as a 3D mesh of an environment. A mesh may comprise a detailed geometric representation of various features including real and virtual objects and surfaces thereof within a particular environment or region of an environment.”); and managing visualization of the 3D point group data on the basis of an installation position of the real world information acquisition device (Geisner: para 70, “The scene mapping engine 306 identifies the position and tracks the movement of real and virtual objects in the volumetric space based on communications with the object recognition engine 192 of the image and audio processing engine 191 and one or more executing applications 166 generating virtual objects.”; also, para 68, “For example, location data such as GPS data from a GPS transceiver 144 on the display device 2 may identify the location of the user.”).
Regarding claim 15, Geisner as modified by Klingler and Seefeldt discloses the one or more non-transitory computer storage media of claim 8, wherein storing 3D point group data of a real space acquired from the real world information acquisition device in the storage comprises: acquiring the 3D point group data from the environmental sensor (Geisner: para 70, “Image and depth data from multiple perspectives can be received in real time from other 3D image capture devices 20 under control of one or more network accessible computer systems 12 or from at least one other personal A/V apparatus 8 in the environment. Depth images from multiple perspectives may be combined based on a view independent coordinate system for describing an environment (e.g. an x, y, z representation of a room, a store space, or a geofenced area) for creating the volumetric or 3D mapping.”); storing the acquired 3D point group data (Geisner: para 66, “The map can be stored in the view independent coordinate system in a storage location (e.g. 324) accessible as well by other personal A/V apparatus 8, other computer systems 12 or both, be retrieved from memory and be updated over time as one or more users enter or re-enter the environment.”); and acquiring an installation position of the real world information acquisition device (Geisner: para 49, “Inside, or mounted to temple 102, are an ear phone 130 of a set of ear phones 130, inertial sensors 132, one or more location or proximity sensors 144, some examples of which are a GPS transceiver, an infrared (IR) transceiver, or a radio frequency transceiver for processing RFID data.”; also, para 68, “For example, location data such as GPS data from a GPS transceiver 144 on the display device 2 may identify the location of the user.”).
Regarding claim 16, Geisner as modified by Klingler and Seefeldt discloses the one or more non-transitory computer storage media of claim 8, wherein analyzing sound information data of ambient sound acquired from the real world information acquisition device comprises: acquiring the sound information data from the acoustic sensor (Geisner: para 34, “The frame 115 includes a nose bridge portion 104 with a microphone 110 for recording sounds and transmitting audio data to control circuitry 136”); and managing an analysis rule of the sound information data (Geisner: para 78, “Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car. Additionally, pre-generated 3D maps of a location can provide an audio index of sounds of objects fixed in the location or which enter and leave the location on a regular basis, e.g. train and bus sounds.”).
Regarding claim 17, Geisner as modified by Klingler and Seefeldt discloses the one or more non-transitory computer storage media of claim 16, wherein determining whether a type of the sound information data corresponds to an artificial sound or a natural sound comprises: analyzing the acquired sound information data on the basis of the analysis rule (Geisner: para 78, “Sound recognition software engine 194 of the 3D audio engine identifies audio data from the real world received via microphone 110 for application control via voice commands and for environment and object recognition.”); and calculating the type of the sound information data (Geisner: para 78, “Based on a sound library 312, the engine 304 can identify a sound with a physical object, e.g. a horn sound associated with a certain make or model of car.”).
Regarding claim 19, Geisner as modified by Klingler and Seefeldt discloses the one or more non-transitory computer storage media of claim 8, further comprisingchange rule of an attribute of the 3D point group data; and changing the attribute of the 3D point group data in the storage on the basis of the change rule.
However, in a similar field of endeavor, Seefeldt discloses managing a change rule of an attribute of the 3D point group data (col 3, “for each animation control signal, the parameters governing the exact behavior of the functions M(s.sub.k,n), C(s.sub.k,n), and b(s.sub.k,n) may be randomized in order to introduce visual variety. For example, in the case of FIG. 2, the spiral velocity may be chosen randomly between some preset range of values.”); and changing the attribute of the 3D point group data in the storage on the basis of the change rule (col 2, “The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's invention, as combined with Klingler's natural-versus-artificial sound classifier and Seefeldt's audio-triggered attribute transformation, with the further features of Seefeldt's invention of maintaining a managed set of parametric change rules and separately applying those rules to change the attribute of the rendered surface. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Seefeldt teaches that "the parameters governing the exact behavior of the functions M(s.sub.k,n), C(s.sub.k,n), and b(s.sub.k,n) may be randomized in order to introduce visual variety," establishing that the change rule is a managed, configurable set of parameters, which corresponds to the recited managing of a change rule. Second, Seefeldt teaches that "The mapped image is then translated and/or deformed by transformation functions triggered by events in an audio or visual signal," which is the act of applying that managed rule to change the attribute of the point cloud. Third, separating the management of the parametric rule from the per-frame application of that rule mirrors the structure Seefeldt itself describes and allows the rule parameters to be maintained independently of the per-frame transformation of Geisner's stored point cloud.
Regarding claim 20, Geisner as modified by Klingler and Seefeldt discloses the one or more non-transitory computer storage media of claim 8, wherein constructing the virtual space video using the adjusted 3D point group data comprises: visualizing the 3D point group data in which the attribute information is modified to generate video data of a virtual space (Geisner: para 72, “In some examples, a 3D mapping, whether it be a depth map generated by front facing cameras 113 including a display field of view, a 3D mapping of an environment or a location in a view independent coordinate system, or somewhere in between, may be modeled as a 3D mesh of an environment. A mesh may comprise a detailed geometric representation of various features including real and virtual objects and surfaces thereof within a particular environment or region of an environment.”); and managing visualization of the 3D point group data on the basis of an installation position of the real world information acquisition device (Geisner: para 70, “The scene mapping engine 306 identifies the position and tracks the movement of real and virtual objects in the volumetric space based on communications with the object recognition engine 192 of the image and audio processing engine 191 and one or more executing applications 166 generating virtual objects.”; also, para 68, “For example, location data such as GPS data from a GPS transceiver 144 on the display device 2 may identify the location of the user.”).
Claim(s) 4, 12, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Geisner et al. (U.S. Pub. No. 20130083173) in view of Seefeldt et al. (U.S. Doc. No. 6448971) and Klingler et al. (U.S. Pub. No. 20200090644), further in view of Soemo et al. (U.S. Pub. No. 20130060571) as applied to claim(s) 1 above, and further in view of Soemo et al. (U.S. Pub. No. 2013/0060571).
Regarding claim 4, Geisner as modified by Klingler and Seefeldt discloses the information processing device according to claim 1, disclose the sound information analysis unit comprises an interface function with an AI system having a sound information analysis function, and a function of transmitting the acquired sound information data to the AI system and receiving a result of analysis of the sound information data by the AI system.
However, in a similar field of endeavor, Klingler discloses the sound information analysis unit comprises an interface function with an AI system having a sound information analysis function (para 45, “the natural versus artificial sound discriminator includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision.”; also, para 19, “the classifier 26 makes a binary classification and determines whether the audio signal relates to a natural sound versus an artificial sound (also referred to as a playback sound).”), and a function of transmitting the acquired sound information data to the AI system and receiving a result of analysis of the sound information data by the AI system (paragraph 25, “In one embodiment, application server 150 may receive an audio file and one or more keywords from computing environment 12. The application server 150 may identify one or more speech sounds within the audio file associated with the one or more keywords. Subsequently, application server 150 may adapt a cloud-based speech recognition technique based on the one or more speech sounds, perform the cloud-based speech recognition technique on the audio file, and transmit one or more words identified within the audio file to computing environment 12.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's invention of a personal augmented reality apparatus, as modified by Seefeldt's audio-triggered attribute transformation and Klingler's machine-learning sound classifier, with the features of Soemo's invention of an integrated local and cloud-based architecture in which a local computing environment transmits an audio file to a remote application server for analysis and receives the analysis result back, so that Klingler's machine-learning sound classifier is provided as the AI system to which the sound information analysis unit transmits the acquired sound information data and from which it receives the result of the sound analysis. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Klingler expressly teaches that "the natural versus artificial sound discriminator includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision," establishing that Klingler's sound classifier is an artificial-intelligence system having a sound information analysis function, namely the function of analyzing the sound information data to determine whether it corresponds to a natural sound or an artificial sound. Second, Soemo expressly teaches that the "application server 150 may receive an audio file and one or more keywords from computing environment 12" and "transmit one or more words identified within the audio file to computing environment 12," which provides a concrete client-server interface for transmitting the acquired sound information data to a remote analysis system and receiving the result of analysis back. Third, implementing Klingler's machine-learning sound classifier as a remote artificial-intelligence system accessed through Soemo's transmit-and-receive interface allows the computationally intensive sound classification to run on a server more capable than Geisner's wearable apparatus, because Soemo teaches that "cloud-based speech recognition may leverage large-scale machine learning and larger acoustic models" than a local device, while preserving the real-time return of the artificial-versus-natural result needed to drive the attribute change of the point cloud.
Regarding claim 12, Geisner as modified by Klingler and Seefeldt discloses the information processing method of claim 7, further comprising: acquired sound information data to an Al system; and receiving a result of analysis of the sound information data by the Al system.
However, in a similar field of endeavor, Soemo discloses transmitting the acquired sound information data (para 25, “In one embodiment, application server 150 may receive an audio file and one or more keywords from computing environment 12.“), and receiving a result of analysis of the sound information data (para 25, “Subsequently, application server 150 may adapt a cloud-based speech recognition technique based on the one or more speech sounds, perform the cloud-based speech recognition technique on the audio file, and transmit one or more words identified within the audio file to computing environment 12.”), while Klingler discloses an Al system (para 45, “the natural versus artificial sound discriminator includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision.”; also, para 25, “one embodiment, application server 150 may receive an audio file and one or more keywords from computing environment 12.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's method, as modified by Seefeldt's audio-triggered attribute transformation and Klingler's machine-learning sound classifier, with the features of Soemo's invention of transmitting an audio file from a local computing environment to a remote application server and receiving the analysis result back, so that the acquired sound information data is transmitted to Klingler's machine-learning sound classifier as a remote AI system and the result of the sound analysis is received back. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Klingler expressly teaches that the sound classifier "includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision," establishing the AI system to which the acquired sound information data is transmitted and which performs the analysis of the sound information data. Second, Soemo expressly teaches that the "application server 150 may receive an audio file and one or more keywords from computing environment 12," providing the act of transmitting the acquired sound information data to that AI system, and that the server may "transmit one or more words identified within the audio file to computing environment 12," providing the act of receiving the result of analysis back. Third, offloading Klingler's machine-learning sound classification from Geisner's wearable apparatus to a remote server accessed through Soemo's interface enables larger and more capable models than can run on the local device, because Soemo teaches that "cloud-based speech recognition may leverage large-scale machine learning and larger acoustic models," while preserving the real-time return of the result needed to update the point cloud attributes.
Regarding claim 18, Geisner as modified by Klingler and Seefeldt discloses the one or more non-transitory computer storage media of claim 8, further comprising: acquired sound information data to an Al system; and receiving a result of analysis of the sound information data by the Al system.
However, in a similar field of endeavor, Soemo discloses transmitting the acquired sound information data (para 25, “In one embodiment, application server 150 may receive an audio file and one or more keywords from computing environment 12.”), and receiving a result of analysis of the sound information data (para 25, “Subsequently, application server 150 may adapt a cloud-based speech recognition technique based on the one or more speech sounds, perform the cloud-based speech recognition technique on the audio file, and transmit one or more words identified within the audio file to computing environment 12.”) while Klingler discloses an Al system (para 45, “the natural versus artificial sound discriminator includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision.”; also, para 19, “the classifier 26 makes a binary classification and determines whether the audio signal relates to a natural sound versus an artificial sound (also referred to as a playback sound).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Geisner's invention, as modified by Seefeldt's audio-triggered attribute transformation and Klingler's machine-learning sound classifier, with the features of Soemo's invention of transmitting an audio file from a local computing environment to a remote application server and receiving the analysis result back, so that the acquired sound information data is transmitted to Klingler's machine-learning sound classifier as a remote AI system and the result of the sound analysis is received back. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Klingler expressly teaches that the sound classifier "includes a classifier 26, employing a neural network or other suitable machine or supervised learning, whose output may be a natural vs. artificial decision," establishing the AI system to which the acquired sound information data is transmitted and which performs the analysis of the sound information data. Second, Soemo expressly teaches that the "application server 150 may receive an audio file and one or more keywords from computing environment 12," providing the operation of transmitting the acquired sound information data to that AI system, and that the server may "transmit one or more words identified within the audio file to computing environment 12," providing the operation of receiving the result of analysis back. Third, offloading Klingler's machine-learning sound classification from Geisner's wearable apparatus to a remote server accessed through Soemo's interface enables larger and more capable models than can run on the local device, because Soemo teaches that "cloud-based speech recognition may leverage large-scale machine learning and larger acoustic models," while preserving the real-time return of the result needed to update the point cloud attributes.
Response to Arguments
Applicant's arguments filed May 27, 2026 have been fully considered.
Regarding the remarks on page 9, applicant's amendment of claim 8 to recite "non-transitory computer storage media" is persuasive. The recited medium no longer encompasses software per se or a propagating transitory signal, and the medium falls within a statutory category. The 35 U.S.C. 101 rejection of claim 8 is withdrawn.
Regarding the rejections under 35 U.S.C. 103, Applicant's arguments have been considered but are moot in view of Applicant's amendment, which necessitated a new ground of rejection. Independent claim 1 was amended to add three features: determining whether the sound information data corresponds to an artificial sound or a natural sound; in response to the natural sound, adjusting a position of a first subset of the 3D point group data; and in response to the artificial sound, adjusting a color of a second subset of the 3D point group data.
The new ground of rejection addresses each of these features. First, the determination of whether the sound corresponds to an artificial sound or a natural sound is taught by Klingler, newly applied to address this limitation, which discloses that "the classifier 26 makes a binary classification and determines whether the audio signal relates to a natural sound versus an artificial sound." Second, adjusting a color of a second subset of the 3D point group data in response to the artificial sound is taught by Seefeldt, which discloses that "the color of each vertex is changed in response to detected triggering events." Third, adjusting a position of a first subset of the 3D point group data in response to the natural sound is recited in the alternative with the color adjustment, joined by "or," and under MPEP 2111.04 and Ex parte Schulhauser need not be shown in the prior art, because the claim is satisfied by the artificial-sound color adjustment.
Independent claims 7 and 8, the corresponding method and computer storage media claims, were amended to recite the same three features and are subject to the same new ground of rejection for the same reasons. The new ground of rejection over Geisner in view of Seefeldt and Klingler is set forth above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights