DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after a Notice of Allowance. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the Notice of Allowance of the previous office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 10/29/25 is being considered, wherein claims 1, 7 and 13 have been amended and claims 6, 12 and 18 have been canceled. Claims 1-5, 7-11 and 13-17 are pending.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-5, 7-11 and 13-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Pardeshi et al. (pub. no. 20220215232) in view of Kirchmayer et al. (pub. no. 20230334754).
Regarding claim 1, Pardeshi discloses a method for generating a three-dimensional (3D) content moment from a video game, comprising: capturing two-dimensional (2D) gameplay video generated from a session of a video game (“In at least one embodiment, a system 200 for generating spectator views can be utilized as illustrated in FIG. 2. In at least one embodiment, a set of instances 202, 204, 206 of player video can be provided as input to a view generator 210 system, service, module, component, or process. In at least one embodiment, these instances can include video clips, files, segments, or streams representing unique perspectives of players or other individuals or entities in a photographic, graphic, digital, virtual, or video game environment. In at least one embodiment, there may be any number of player views provided as input, as may be limited by factors such as game or resource limitations”, [0056];
“In at least one embodiment, a process for generating a spectator view can be performed 400 as illustrated in FIG. 4A. In at least one embodiment, a set of video data is obtained 402 for individual actors in an environment, such as players in an online game session, along with map data for that environment. In at least one embodiment, this can include a set of video clips or streams representing unique perspectives of individual players or actors in those respective video clips”, [0064]);
analyzing the 2D gameplay video to determine 3D geometry of a scene depicted in the 2D gameplay video (“In at least one embodiment, this data can be provided as input to a view generator 210. In at least one embodiment, view generator 210 includes at least four modules. In at least one embodiment, a first module is a motion classifier module 212. In at least one embodiment, pre-processed videos, or other instances of video data, can be motion-labelled for motion classification. In at least one embodiment, video pre-processing can provide normalization, as well as a selection of resolution that may be appropriate for a given system for a given frame rate. In at least one embodiment, this motion classification can be performed using a three-dimensional convolutional neural network (3D-CNN) with attention capability. In at least one embodiment, input video clips can be processed by this model on a frame-by-frame basis for a set of subsequent frames to determine motion. In at least one embodiment, labels can be applied to an original frame based at least in part upon confidence of this model. In at least one embodiment, a result can be individual video clips with approximate overall motion labels corresponding to each instance of time. In at least one embodiment, such a CNN model can be trained using supervised learning on a labelled dataset containing aggregated motion of objects at a center of a frame (e.g., for a third-person POV) or at a frame level (e.g., for a first person POV). In at least one embodiment, an actor may appear to move in an environment in a third person view, where that avatar may be treated as an object of motion. In at least one embodiment, an actor may appear with a first person perspective, where an entire space or view moves instead of a specific object. In at least one embodiment, a 3D-CNN can classify each of these types of frames accordingly, and can provide a video clip with motion labels for each frame of that clip.
In at least one embodiment, this output can be passed to an autoencoder module, such as may include an intersecting variational auto-encoder (VAE) 214. In at least one embodiment, this auto-encoder can be a model that encodes scene-level features of each clip, along with a direction of motion for every instance of time (e.g., each frame) in that clip. In at least one embodiment, this module can maintain memory for mapping between scene-level features and their placements in latent space, as well as instances of time with direction of motion information. In at least one embodiment, a latent space for every clip can be constrained to represent a location corresponding to specific features in that scene. In at least one embodiment, latent spaces of individual VAEs, that may each represent one or more actors, can be projected onto each other to determine overlaps. In at least one embodiment, overlaps in a latent space can represent two actors being in a same or similar location, although not necessarily at a same or similar time. In at least one embodiment, sampling from areas where a latent space overlaps might otherwise map to a time instant and direction of motion for a given player using memory of this model. In at least one embodiment, both an encoder and a decoder of this VAE may be trained to build a latent space of visual features, with memory for time and motion corresponding to each instance. In at least one embodiment, this decoder may be detached once training is complete. In at least one embodiment, such a model may not always produce a 1:1 mapping between memory and latent space due to its “variational” nature, but can be approximate in range. In at least one embodiment, instead of an encoder of this VAE simply mapping features into a latent space, this encoder can keep track of where different types of features are moving in different spaces, based on additional information such as time and direction of motion. In at least one embodiment, this time and direction of motion data can be encoded into this latent space for each actor. In at least one embodiment, this information may overlap for certain periods of time. In at least one embodiment, an actor may remain in one area for a short period of time, then move to another area, where that latent space may then correspond to a different location. In at least one embodiment, a separate latent space may be maintained for each actor in a game session or experience. In at least one embodiment, an overlap between two or more latent spaces can provide a strong indication that these actors interacted in some way, or at least were in similar locations at similar or different points in time. In at least one embodiment, these interactions can be used to correlate image data from different video feeds, files, or streams.
In at least one embodiment, this output can then be provided as input to a trajectory mapping module, or trajectory mapper 216. In at least one embodiment, this mapper can sample from one or more locations of overlaps for each corresponding VAE, and can reference from memory associated with those instances. In at least one embodiment, time and motion of individual players can be obtained at this point in time. In at least one embodiment, a trajectory for each actor (going back in time) can be determined with respect to, and potentially plotted onto, a reference map based on these motion-classified clips. In at least one embodiment, this can be just programmatic logic that interacts with intersecting VAE 214. In at least one embodiment, this mapping can provide a set of 2D trajectories to use to enable a generator to build an output image.
In at least one embodiment, this information (e.g., actor trajectories, latent spaces, and a reference map) can be provided as input to a generator module 218, as may include a dual-stage generative adversarial network (GAN). In at least one embodiment, these trajectories can be sampled from respective intersecting VAEs 214 and mapped by trajectory mapper 216. In at least one embodiment, during a first stage this GAN can generate frames corresponding to each location in every trajectory and instance of time by sampling from individual VAEs. In at least one embodiment, overlaps can be accounted for inherently, based at least in part upon trajectories using multi-sampling from latent spaces of respective VAEs. In at least one embodiment, positional metadata inputs can account for an extent of motion per instance of time for each player. In at least one embodiment, an output of this first stage can be a spectator representation of an aggregated scene at that location and instance of time, corresponding to a change in point of view (POV). In at least one embodiment, training for this first stage of this GAN can utilize trajectories and associated samples from these latent spaces that serve as a grooming condition for a generator of this GAN. In at least one embodiment, points from this reference map can serve as a noise factor, inducing variety in sampling. In at least one embodiment, a discriminator of this GAN can be trained on spectator POV clips.
In at least one embodiment, during a second stage this GAN can resample from this first stage generator for close neighbors of every location, in order to obtain one or more alternatives for every frame. In at least one embodiment, these variants can be used to average and fit onto a representation, such as a cube map representation. In at least one embodiment, probabilistic distributions can be utilized to determine these values. In at least one embodiment, an extrapolation can then be performed to fill or complete image data for missing locations or directions. In at least one embodiment, this extrapolation can be performed by this GAN using other information in these views based at least in part upon training of this GAN. In at least one embodiment, a resulting image provided by this GAN may fit a cube map representation (or other 360 degree representation) in various ways, depending at least in part upon output from this GAN 218. In at least one embodiment, there may be multiple images or video frames, with those frames being assumed to fit into a 360 degree representation. In at least one embodiment, one side (or face) of a cube map can be assumed to be at 100%, with overlaps in other sides that may need to be extrapolated. In at least one embodiment, generated output may fit a front face with 100%, a top face with 20%, a bottom face with 30%, a left face with 10%, a right face with 25%, and a back face with 0%. In at least one embodiment, these fit factors can be determined dynamically by this model based at least in part upon averaging across multiple samples. In at least one embodiment, training can be performed with multiple input examples for each case, at least to account for averaging. In at least one embodiment, a final output can be a spectator image 220 or video that is time bound and position bound in this map. In at least one embodiment, this can be visualized using a tool that sets a range for each of these parameters. In at least one embodiment, an output can be a spectator view of this environment with a 360-degree extrapolation of details from a determined third-person view. In at least one embodiment, this output can be time-bound and position-bound in this environment map. In at least one embodiment, a spectator view may be limited to these points, where changes can relate to zoom and orientation, but in at least one embodiment, a virtual camera may be allowed to pan or translate within this environment as well. In at least one embodiment, a spectator can specify a time and location in a map for a game session or experience, and can obtain a 360 degree spectator view for that time and location.
In at least one embodiment, a set of images 302, 304, 36, 308, 310, 312 can be obtained or generated for a location based at least in part upon correlated video feed information. In at least one embodiment, image data may represent image information for at least a subset of a 360 degree view, and a GAN can extrapolate or otherwise fill in holes or gaps in these images to generate image data for a full 360-degree view. In at least one embodiment, data for these images can be used to generate a 360-degree representation, such as a cube map 310. In at least one embodiment, and as illustrated, cube map 310 is a representation of a scene that, if one were to fold this cube map into a box and stand at a middle coordinate inside this folded box, would provide a quasi-360 degree view of this scene. In at least one embodiment, this cube map 310 can be transformed to a 360-degree representation 312 from which a specific point of view can be selected and rendered for viewing on a display. In at least one embodiment, this may also be a spherical panoramic representation that is compatible with various AR and VR applications, as may involve at least some amount of post-processing to generate using cube map 310. In at least one embodiment, any gaps in view directions from these input images can be filled in during generation of cube map 310. In at least one embodiment, cube map 310 will be a full representation of a respective scene for a determined location and point in time, and a transformation into a spherical (or other) representation will not involve an additional filling or generation of new image content, other than corresponds directly to this transformation between representations”, [0058] – [0063];
“In at least one embodiment, additional metadata may also be received 404 that provides information about this session, this environment, or these actors, among other such options. In at least one embodiment, this video data can be analyzed 406 to classify motion determined for each frame, or at least a subset of frames from a set of video clips. In at least one embodiment, this motion and feature data can be encoded 408 into one or more latent spaces, such as a latent space for each actor represented in this video data. In at least one embodiment, a set of actor trajectories through this environment can be determined 410 using this motion and feature data relative to relevant map data for this environment”, [0064]);
using the 3D geometry of the scene to generate a 3D video asset of a moment that occurred in the gameplay video; and storing the 3D video asset to a user account (“In at least one embodiment, this spectator view 130 can be generated to provide a view that is not tied to a specific player, or player point of view. In at least one embodiment, a view can be generated that may be more of an overhead or far away view, for example, which may allow for multiple players to be concurrently visible in a spectator view. In at least one embodiment, such a view may only show a single player avatar, but may provide a better view of an interaction of that player or surroundings of that player. In at least one embodiment, a view may be generated that, for at least a period of time, may not show any players but may show other occurrences in a game session. In at least one embodiment, a spectator view 130 can be provided that includes avatars for all three players 102, 112, 122 in order to view actions, and interactions, of those players. In at least one embodiment, this view can be selected to provide a view of all players, or specific types of interactions between players. In at least one embodiment, this view may be selected based on specific selection criteria of a view generation application, or may be controlled at least in part based upon user input. In at least one embodiment, a spectator view can be generated that is a third-person, 360-degree view of at least a portion of a gameplay level, map, or world, where that view can be generated based at least in part upon video or image data provided for individual players or groups of players “, [0047];
“In at least one embodiment, data may be obtained and processed for an entire session in order to provide one or more spectator views for any point in that session. In at least one embodiment, spectator views may only be generated for specific events or occurrences in a session. In at least one embodiment, this may include specific types of events in a game session instead of an entire game session. In at least one embodiment, this can allow for spectator views in situations where player is killed, a player obtains an achievement, or another such type of action, instead of providing spectator views when players are merely walking or exploring a level in a session. In at least one embodiment, this may be used to provide a type of highlight or replay during live gameplay, where when an event of interest occurs a spectator can access a generated spectator view in order to obtain a different view of that occurrence”, [0052];
“In at least one embodiment, one or more spectator views can be generated 412 based at least in part upon overlaps or interactions determined using these trajectories. In at least one embodiment, these spectator views can include one or more images generated for a location and point in time from a session. In at least one embodiment, additional sampling can be performed 414 for these spectator views in order to attempt to fill any gaps in image data or improve a quality of image data for any portions of these spectator views. In at least one embodiment, a three-hundred sixty degree representation can be generated 416 for this spectator view, which can enable a spectator to perform actions such as to rotate or zoom a virtual camera associated with that spectator view. In at least one embodiment, this representation of this spectator view can be provided 418 for use in displaying this spectator view through a display or other presentation mechanism. In at least one embodiment, actions such as a rotate or zoom may not require generation and providing of a new or updates spectator view representation, while actions such as a translation or other camera selection may require generation or providing of a new or updated spectator view representation”, [0064]).
Regarding claim 1, it is noted that Pardeshi does not explicitly disclose determining a texture, a shading, or a lighting of the scene, and incorporating that texture, shading, or lighting in the 3D video asset. Kirchmayer however, teaches determining a texture, a shading, or a lighting of the scene, and incorporating that texture, shading, or lighting in the 3D video asset (“More particularly, in FIG. 6A, a stream receiver subsystem 602 receives the 2D video data from the first video data stream 116 and outputs that 2D video data to an eye area deocclusion subsystem 604. The eye area deocclusion subsystem 604 sends 2D video data in parallel to a 2D-to-3D reconstruction subsystem 606 and to a texture reconstruction subsystem 608. The 2D-to-3D reconstruction subsystem 606 outputs 3D data, and the texture reconstruction subsystem 608 outputs texture data in 2D, and both the texture data and 3D data are sent to a 3D processing subsystem 610. The 3D processing subsystem 610 sends 3D and texture data, together with related data as discussed below, to a stream receiver subsystem 614 on the first display device 112”, [0067];
“In at least some example embodiments, the texture reconstruction subsystem 608 reconstructs color information for portions of the holographic projection 108 that are not depicted in the 2D data in the first video data stream 116. The artificial neural network used for texture reconstruction may be based on the CNN described in Isola. The CNN receives as input the 2D image data from the camera 124 and outputs data representing 2D color texture for the coloring of the 3D volume output by the 2D-to-3D reconstruction subsystem 606. The output of texture reconstruction subsystem 608 is given in suitable coordinates, such as cylindrical, spherical, or another suitable 3D texture space to enable the first display device 112 to generate the holographic projection 108”, [0076]).
Exemplary rationales that may support a conclusion of obviousness include combining prior art elements according to known methods to yield predictable results. Here both Pardeshi and Kirchmayer are directed to systems that generate 3d video assets from a 2d video. To add the texture reconstruction subsystem of Kirchmayer to the Pardeshi invention would be to combine a prior art element according to a known method to yield predictable results. Therefore, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the claimed invention to modify Pardeshi to use the texture reconstruction subsystem of Kirchmayer. To do so would increase the verisimilitude of the of the 3d video thereby increasing the perceived entertainment value of the system.
Regarding claim 2, Pardeshi discloses identifying and tracking objects depicted in the 2D gameplay video ([0058]).
Regarding claim 3, Pardeshi discloses providing an interface that renders a view of the 3D video asset for presentation on a display (“In at least one embodiment, one or more spectator views can be generated 412 based at least in part upon overlaps or interactions determined using these trajectories. In at least one embodiment, these spectator views can include one or more images generated for a location and point in time from a session. In at least one embodiment, additional sampling can be performed 414 for these spectator views in order to attempt to fill any gaps in image data or improve a quality of image data for any portions of these spectator views. In at least one embodiment, a three-hundred sixty degree representation can be generated 416 for this spectator view, which can enable a spectator to perform actions such as to rotate or zoom a virtual camera associated with that spectator view. In at least one embodiment, this representation of this spectator view can be provided 418 for use in displaying this spectator view through a display or other presentation mechanism. In at least one embodiment, actions such as a rotate or zoom may not require generation and providing of a new or updates spectator view representation, while actions such as a translation or other camera selection may require generation or providing of a new or updated spectator view representation”, [0064]).
Regarding claim 4, Pardeshi discloses the interface enables adjustment of a perspective of the view of the 3D video asset ([0064]).
Regarding claim 5, Pardeshi discloses the 3D video asset defines a 3D content model of the scene depicted in the 2D gameplay video ([0063]).
Claims 7-11 are directed to an article of manufacture containing code that implements the methods of claims 1-5 respectively and are rejected for the same reasons as claims 1-5 respectively.
Claims 13-17 are directed to systems that implement the methods of claims 1-5 respectively and are rejected for the same reasons as claims 1-5 respectively.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LAWRENCE STEFAN GALKA whose telephone number is (571)270-1386. The examiner can normally be reached M-F 6-9 & 12-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Lewis can be reached at 571-272-7673. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LAWRENCE S GALKA/Primary Examiner, Art Unit 3715