DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The objection to the drawings and the specifications have been withdrawn in view of applicant’s amendments filed on 04/29/2026
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Thurston et al. (U.S. Pub. No. 20230186550) in view of Schmirler et al. (U.S. Pub. No. 20180131907).
Regarding claim 1, Thurston discloses a computer-implemented method comprising (para 9, “One general aspect includes a computer-implemented method of generating a virtual scene rendering usable in a captured scene"): providing a scene model that is a virtual representation of a scene based on depth and color data captured for the scene (para 49, “a virtual space 150 that may for example be populated with digital scenes, objects, characters, environments, etc.”; also, para 35, “Each entry in the pixel image value array might comprise a pixel color value, an optional alpha value, a depth value or a depth range, and an object identifier identifying which object contributes that color/alpha at the specified depth.”; also, para 38, “the live action scene may also be captured by other sensors and/or sensing devices, including optical sensing devices, depth sensors and/or ranging sensors (e.g., LiDAR), and the like.”); miniaturized version of the scene model by scaling the scene model to provide a diorama of the scene; setting a virtual camera with respect to the scene model to provide a perspective view of the diorama; and causing the diorama of the scene to be outputted at the perspective view on an output device.
However, in a similar field of endeavor, Schmirler discloses creating a miniaturized version of the scene model by scaling the scene model to provide a diorama of the scene (para 50, “the system can render a scaled down view of the factory floor area, which affords the user an external overview of the area.”; also, para 92, “the external view generated by VR/AR presentation system 302 renders the industrial area as a virtual scale model of the area, and allows the user to move around and interact with the scaled version of the area”); setting a virtual camera with respect to the scene model to provide a perspective view of the diorama (para 10, “This allows the user to virtually walk around the model and view aspects of the production area from an external overview perspective”; also, para 91, “the direction and angle of the viewing perspective of the VR presentation is a function of the first user's location and orientation”); and causing the diorama of the scene to be outputted at the perspective view on an output device (para 50, “the holographic views can be delivered to a wearable visualization computer, which renders the 3D view as a function of the user's current location and/or orientation.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of a scene model that is a virtual representation of a scene based on captured depth and color data, rendered from a virtual camera viewpoint to a display, with the features of Schmirler's invention of rendering a captured area as a scaled-down three-dimensional scale model viewed from an external overview perspective. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Thurston builds a virtual scene from a real scene captured with depth sensors and color image data and renders that scene from a virtual camera viewpoint to a display, but presents the scene only at full scale. Schmirler teaches that the system "can render a scaled down view of the factory floor area, which affords the user an external overview of the area" and "renders the industrial area as a virtual scale model of the area," which is a miniaturized version of the scene produced by scaling, that is, a diorama. Second, Schmirler further teaches that the user can "virtually walk around the model and view aspects of the production area from an external overview perspective," where "the direction and angle of the viewing perspective of the VR presentation is a function of the first user's location and orientation," and that "the holographic views can be delivered to a wearable visualization computer," so the scaled-down diorama is provided at a perspective view and output on a display. Third, applying Schmirler's scaled-down scale model to Thurston's captured scene model, using Thurston's virtual camera viewpoint and display to render and output the diorama, lets a user apprehend and move around the whole scene at a glance, addressing the limitation of Thurston's full-scale presentation that shows only the portion of the scene framed by the camera, so a person of ordinary skill would have produced and displayed the diorama as taught by Schmirler with a predictable and reliable result.
Regarding claim 2, Thurston as modified by Schmirler discloses the computer-implemented method of claim 1, wherein said causing comprises generating augmented video data comprising one or more composited video frames with the diorama at the perspective view (Thurston: para 51, “a compositor may correspond to hardware and/or software that may combine image data of a captured live action scene and a rendered scene to form composited imagery.”).
Regarding claim 3, Thurston as modified by Schmirler discloses the computer-implemented method of claim 2, wherein said generating augmented video data comprises compositing one or more video frames provided by a video camera and the diorama to provide the augmented video data (Thurston: para 51, “a compositor may correspond to hardware and/or software that may combine image data of a captured live action scene and a rendered scene to form composited imagery.”; also, para 17, “pixels representing the given virtual scene element in a composite image, captured by the camera, of the stage environment and the virtual scene display.”).
Regarding claim 4, Thurston as modified by Schmirler discloses the computer-implemented method of claim 2, wherein the virtual camera is further set based on camera viewpoint data for the video camera to specify the perspective view of the diorama relative to the scaled scene model (Thurston: para 17, “the depth value corresponds to a virtual distance in the virtual scene between the given virtual scene element and a virtual camera viewpoint located at the camera position”; also, para 10, “determining the camera position includes reading data from camera position sensors placed on the camera.”).
Regarding claim 5, Thurston as modified by Schmirler discloses the computer-implemented method of claim 4, the computer-implemented method of claim 4, top down view, an oblique view, or a slanted view of the diorama.
However, in a similar field of endeavor, Schmirler discloses wherein the perspective view includes a top down view, an oblique view, or a slanted view of the diorama (para 100, “This allows the user to virtually walk around the model and view aspects of the production area from an external overview perspective.”; also, para 91, “the direction and angle of the viewing perspective of the VR presentation is a function of the first user's location and orientation.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of rendering a captured scene model from a virtual camera with the features of Schmirler's invention of viewing the scaled model from an external overview perspective at a user-controlled angle. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Schmirler teaches allowing the user to "virtually walk around the model and view aspects of the production area from an external overview perspective," which is an oblique or slanted view of the diorama taken from an external vantage rather than head-on. Second, Schmirler teaches that "the direction and angle of the viewing perspective of the VR presentation is a function of the first user's location and orientation," so the user can select the angle from which the diorama is viewed, including a slanted or overhead angle. Third, providing a user-controlled external overview angle lets a viewer inspect the diorama of the captured scene from an angle, which gives a more complete understanding of the spatial layout than a single fixed viewpoint, so a person of ordinary skill would have applied Schmirler's external overview perspective to the perspective view of Thurston's diorama with a predictable result.
Regarding claim 6, Thurston as modified by Schmirler discloses the computer-implemented method of claim 4, wherein the camera is a first video camera, the output device is a first output device, the virtual camera is a first virtual camera, and the perspective view is a first perspective view, and the method further comprising: setting a second virtual camera in the virtual environment with respect to the scene model to provide a second perspective view of the diorama based on camera viewpoint data for a second video camera (Thurston: para 53, “A second camera, second camera position, or second real-world camera 120b has a viewing frustum 130b and corresponding virtual viewing frustum 140b.”; also, para 56, “switching to a different camera view may require updating images on the display wall 102 to account for a different virtual camera position and orientation.”); and diorama of the scene at the second perspective to be rendered on a second output device.
However, in a similar field of endeavor, Schmirler disclose causing the diorama of the scene at the second perspective to be rendered on a second output device (para 90, “the locations and orientations of the human icons 808a and 808b within the VR/AR presentation can be determined based on location and orientation data 606 received by VR/AR presentation system 302 from the wearable appliances 206 associated with each user.”; also, para 91, “Rendering component 308 receives location and orientation data 606 generated by the first user's wearable appliance, and renders the presentation on the wearable appliance in accordance with the first user's current location and orientation”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of rendering a second perspective view from a second virtual camera with the features of Schmirler's invention of rendering the presentation to a separate wearable appliance for each user. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Thurston teaches a second camera with its own virtual viewing frustum but renders to a single display wall, and does not render the second perspective to a second output device. Second, Schmirler teaches that the presentation is rendered to "the wearable appliances 206 associated with each user" and that the rendering component "renders the presentation on the wearable appliance in accordance with the first user's current location and orientation," so a second user's wearable appliance is a second output device that renders a second perspective. Third, providing a second output device in this manner lets a second viewer see the second perspective view of the diorama at the same time as the first viewer sees the first perspective view, so a person of ordinary skill would have rendered the second perspective on a second output device as taught by Schmirler with a predictable result.
Regarding claim 7, Thurston as modified by Schmirler discloses the computer-implemented method of claim 6, first and second output devices are mobile devices.
However, in a similar field of endeavor, Schmirler discloses wherein the first and second output devices are mobile devices (para 50, “the holographic views can be delivered to a wearable visualization computer, which renders the 3D view as a function of the user's current location and/or orientation.”; also, para 69, “Visualization component 408 can be configured to render the virtual reality, augmented reality, mixed reality, or video presentations delivered to the wearable appliance 206 by VR/AR presentation system 302.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of outputting a rendering on a display with the features of Schmirler's invention of delivering the rendered presentation to a wearable appliance. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Schmirler expressly teaches that "the holographic views can be delivered to a wearable visualization computer" and that the presentations are "delivered to the wearable appliance 206," which is a mobile device worn and carried by the user. Second, presenting the first and second perspective views on wearable mobile devices lets viewers access the diorama away from a fixed display, which broadens the practical use of Thurston's captured scene without altering how the scene model is rendered. Third, because Schmirler delivers rendered content of the same general kind to such wearable mobile appliances, a person of ordinary skill would have output Thurston's first and second perspective views to mobile devices as taught by Schmirler with a predictable result.
Regarding claim 8, Thurston as modified by Schmirler discloses the computer-implemented method of claim 2, further comprising inserting one or more digital assets into the scene model representative of digital assets to be used in the scene before creating the miniaturized version of the scene model (Thurston: para 49, “a virtual space 150 that may for example be populated with digital scenes, objects, characters, environments, etc.”; also, para 56, “Positioned within virtual space 150, "behind" video wall 102, is a first virtual object or virtual scene element 180, with a given position, size, shape, and orientation within virtual space 150.”).
Regarding claim 9, Thurston as modified by Schmirler discloses the computer-implemented method of claim 8, wherein the one or more digital assets comprises a first digital asset, the method further comprising insert a second digital asset representative of an actor into the scene model, wherein movements of the second digital asset in the scene model are synced to movements of the actor captured for the scene (Thurston: para 49, “a virtual space 150 that may for example be populated with digital scenes, objects, characters, environments, etc.”; also, para 120, “An animation driver generator 1244 might read in live action metadata, such as data representing movements and positions of body parts of a live actor during a live action scene.”; also, para 120, “Animation driver generator 1244 might generate corresponding animation parameters to be stored in animation sequence storage 1238 for use in animating a CGI object. This can be useful where a live action scene of a human actor is captured while wearing mo-cap fiducials (e.g., high-contrast markers outside actor clothing, high-visibility paint on actor skin, face, etc.) and the movement of those fiducials is determined by live action processing system 1222.”; also, para 120, “Animation driver generator 1244 might convert that movement data into specifications of how joints of an articulated CGI character are to move over time.”).
Regarding claim 10, Thurston as modified by Schmirler discloses the computer-implemented method of claim 1, . Thurston does not disclose wherein the diorama is animated to provide a visual representation of the scene from the perspective view.
However, in a similar field of endeavor, Schmirler discloses wherein the diorama is animated to provide a visual representation of the scene from the perspective view (para 100, “rendering component 308 renders a scaled-down version of the production area that can be viewed by the user as if the production area were an animated, interactive scale model.”; also, para 85, “the plant model can define physical dimensions and colors for the asset, as well as any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.)”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of rendering a diorama of the scene from a virtual camera perspective view with the features of Schmirler's invention of presenting the scaled-down scene as an animated scale model. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Schmirler teaches that the rendering component "renders a scaled-down version of the production area that can be viewed by the user as if the production area were an animated, interactive scale model," which animates the diorama for the viewer. Second, Schmirler further teaches that the model can define "any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.)," which provides a visual representation of the scene that changes over time. Third, animating the diorama in this manner gives the viewer a continuous visual representation of the captured scene from the perspective view, improving comprehension of the spatial layout, so a person of ordinary skill would have animated Thurston's diorama as taught by Schmirler with a predictable result.
Claim(s) 11-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Thurston et al. (U.S. Pub. No. 20230186550) in view of Stoski et al. (U.S. Pub. No. 20230196693).
Regarding claim 11, Thurston discloses a computer-implemented method comprising (para 9, “One general aspect includes a computer-implemented method of generating a virtual scene rendering usable in a captured scene”): (para 51, “a compositor may correspond to hardware and/or software that may combine image data of a captured live action scene and a rendered scene to form composited imagery.”) augmented video data to be rendered on an output device (para 37, “a display wall may be used, which may include a display screen (e.g., an LED, LCD, LED LCD, OLED, or the like) that is capable of outputting a rendering of an image or a video.”). Thurston does not disclose receiving waypoint instructions identifying virtual points that define a path for a digital asset for use in a scene, updating a scene model to include the virtual points at locations in the scene model corresponding to locations in the scene to provide a waypoint scene model that stores the path for the digital asset, with the virtual points in the scene based waypoint scene model.
However, in a similar field of endeavor, Stoski discloses receiving waypoint instructions identifying virtual points that define a path for a digital asset for use in a scene (para 8, “the user may define a route or other series of interactions using waypoints defined by pointing to the corresponding locations in the real-world using the controller.”; also, para 73, “A virtual object (e.g., character) may traverse along the route 405, thus allowing the user to control the movement of the virtual object. Said differently, the user can define a path for a virtual object to follow.”), updating a scene model to include the virtual points at locations in the scene model corresponding to locations in the scene to provide a waypoint scene model that stores the path for the digital asset (para 5, “To enable this, the virtual content is positioned using a 3D map (e.g., a mesh or point cloud) of the real-world environment around the user. The 3D map may use a virtual coordinate system with a one-to-one relationship with at least a portion of the real world.”; also, para 73, “A route 405 is formed according to the defined waypoints. In the example of FIG. 4A, the route 405 is formed to intersect all of the waypoints.”; also, para 63, “More specifically, a waypoint is specified on a 3D map of the real world.”; also, para 54, “In one embodiment, the mapping system 327 stores the 3D map along with any semantic/contextual information in the 3D map store 328”), with the virtual points in the scene based waypoint scene model (para 63, “Waypoints may be displayed to the user.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of generating composited augmented video from a scene model and outputting it on a display with the features of Stoski's invention of defining a path for a digital asset using waypoints placed on a three-dimensional map of the scene. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Thurston builds a scene model and composites a rendered scene with captured imagery, but does not provide for placing path-defining points in the scene model. Stoski teaches positioning virtual content using "a 3D map (e.g., a mesh or point cloud) of the real-world environment" having "a one-to-one relationship with at least a portion of the real world," which is a scene model whose locations correspond to locations in the scene. Second, Stoski teaches that a user "can define a path for a virtual object to follow" by selecting waypoints, and that "a route 405 is formed according to the defined waypoints" and stored so that "a virtual object (e.g., character) may traverse along the route," which provides a waypoint scene model that stores the path for the digital asset. Third, incorporating Stoski's waypoint path into Thurston's scene model lets an author control where a digital asset travels within the captured scene and then composite and render that content, which extends Thurston's static scene presentation to controllable, path-driven content with a predictable result.
Regarding claim 12, Thurston as modified by Stoski discloses the computer-implemented method of claim 11, animation of a digital asset between neighboring virtual points of the virtual points, wherein the waypoint scene model is provided with data specifying the animation of the digital asset between the neighboring virtual points of the path.
However, in a similar field of endeavor, Stoski discloses further comprising receiving animation instructions identifying an animation of a digital asset between neighboring virtual points of the virtual points, wherein the waypoint scene model is provided with data specifying the animation of the digital asset between the neighboring virtual points of the path (para 69, “In another example, if two waypoints are established, a line connecting the two waypoints may indicate a movement direction and movement speed for a virtual object in the real world (e.g., the distance between the waypoints determines the movement speed of the virtual object).”; also, para 68, “A route defines a pathway in the real world for a virtual object to move along. The route may have a start point and an end point.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of generating augmented video from a scene model with the features of Stoski's invention of specifying movement of a virtual object between established waypoints. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Stoski teaches that "if two waypoints are established, a line connecting the two waypoints may indicate a movement direction and movement speed for a virtual object," which specifies how a digital asset is animated between neighboring virtual points. Second, Stoski stores this movement specification as part of the route, which "defines a pathway in the real world for a virtual object to move along" with "a start point and an end point," so the waypoint scene model is provided with data specifying the animation between neighboring virtual points of the path. Third, providing per-segment movement data lets the author control the digital asset's animation along each leg of the path within Thurston's scene model, giving fine control over the asset's behavior with a predictable result.
Regarding claim 13, Thurston as modified by Stoski discloses the computer-implemented method of claim 12, wherein said providing comprises generating the augmented video data with composited frames with the virtual points in the scene (Thurston: para 51, “a compositor may correspond to hardware and/or software that may combine image data of a captured live action scene and a rendered scene to form composited imagery.“) and digital asset between the neighboring virtual points along the path.
However, in a similar field of endeavor, Stoski discloses the digital asset between the neighboring virtual points along the path (para 73, “A virtual object (e.g., character) may traverse along the route 405, thus allowing the user to control the movement of the virtual object. Said differently, the user can define a path for a virtual object to follow.”; also, para 69, “In another example, if two waypoints are established, a line connecting the two waypoints may indicate a movement direction and movement speed for a virtual object in the real world (e.g., the distance between the waypoints determines the movement speed of the virtual object).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of generating composited augmented video frames with the features of Stoski's invention of a virtual object traversing a path between waypoints. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Thurston generates composited frames combining a captured scene and a rendered scene but does not depict an asset moving along a path. Stoski teaches that "a virtual object (e.g., character) may traverse along the route 405," and that movement between two waypoints is controlled along the connecting segment, which places the digital asset between the neighboring virtual points along the path. Second, compositing Stoski's traversing virtual object into Thurston's frames shows the digital asset moving along the defined path within the captured scene. Third, depicting the asset between neighboring waypoints in the composited frames produces augmented video that conveys the asset's motion through the scene, which is a predictable extension of Thurston's compositing to dynamic path-driven content.
Regarding claim 14, Thurston as modified by Stoski discloses the computer-implemented method of claim 13, wherein said causing comprises causing the augmented video to be rendered on the output device to provide a visual animation of the digital asset at and/or between the neighboring points (Thurston: para 37, “a display wall may be used, which may include a display screen (e.g., an LED, LCD, LED LCD, OLED, or the like) that is capable of outputting a rendering of an image or a video.”) waypoint scene model and the path.
However, in a similar field of endeavor, Stoski does not disclose based on the waypoint scene model and the path (para 73, “A virtual object (e.g., character) may traverse along the route 405, thus allowing the user to control the movement of the virtual object. Said differently, the user can define a path for a virtual object to follow.”; also, para 68, “A route defines a pathway in the real world for a virtual object to move along. The route may have a start point and an end point.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of rendering augmented video on a display with the features of Stoski's invention of providing augmented reality data of a virtual object traversing a defined route. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Thurston outputs a rendering on a display but does not base the rendered animation on a stored path. Stoski teaches "providing (e.g., to the client device) AR data including the virtual object traversing along the route," where "a route defines a pathway in the real world for a virtual object to move along," so the rendered animation is based on the waypoint scene model and the path. Second, rendering Thurston's augmented video according to Stoski's stored route causes the displayed animation of the digital asset to follow the path defined by the waypoints. Third, basing the visual animation on the stored waypoint scene model ensures the rendered motion of the asset matches the authored path, which is a predictable and reliable result of combining the two teachings.
Claim(s) 15-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Thurston et al. (U.S. Pub. No. 20230186550) in view of Stoski et al. (U.S. Pub. No. 20230196693) and further in view of Schmirler et al. (U.S. Pub. No. 20180131907).
Regarding claim 15, Thurston as modified by Stoski discloses the computer-implemented method of claim 11, further comprising: miniaturized version of the waypoint scene model by scaling the waypoint scene model corresponding to a diorama of the scene; and setting a virtual camera with respect to the scene model to provide a perspective view of the diorama.
However, in a similar field of endeavor, Schmirler discloses creating a miniaturized version of the waypoint scene model by scaling the waypoint scene model corresponding to a diorama of the scene (para 50, “the system can render a scaled down view of the factory floor area, which affords the user an external overview of the area.”; also, para 92, “the external view generated by VR/AR presentation system 302 renders the industrial area as a virtual scale model of the area, and allows the user to move around and interact with the scaled version of the area.”); and setting a virtual camera with respect to the scene model to provide a perspective view of the diorama (para 100, “This allows the user to virtually walk around the model and view aspects of the production area from an external overview perspective.”; also, para 91, “the direction and angle of the viewing perspective of the VR presentation is a function of the first user's location and orientation.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention as modified by Stoski with the features of Schmirler's invention of rendering a captured area as a scaled-down three-dimensional scale model viewed from an external overview perspective. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Thurston as modified by Stoski provides a waypoint scene model storing a path for a digital asset and a virtual camera viewpoint, but presents the scene at full scale. Schmirler teaches that the system "can render a scaled down view of the factory floor area, which affords the user an external overview of the area" and "renders the industrial area as a virtual scale model of the area," which is a miniaturized version of the scene produced by scaling, that is, a diorama. Second, Schmirler further teaches that the user can "virtually walk around the model and view aspects of the production area from an external overview perspective," where "the direction and angle of the viewing perspective of the VR presentation is a function of the first user's location and orientation," so the scaled-down diorama is provided at a perspective view, and Thurston's virtual camera viewpoint provides the mechanism for that perspective view. Third, presenting the waypoint scene as a scaled-down diorama lets a user review the authored path and its waypoints as a whole, which aids in laying out and reviewing the path, so a person of ordinary skill would have produced and displayed the diorama as taught by Schmirler with a predictable result.
Regarding claim 16, Thurston as modified by Stoski and Schmirler discloses the computer-implemented method of claim 15, wherein said augmented video data is provided with the one or more composited video frames with the diorama at the perspective view (Thurston: para 51, “a compositor may correspond to hardware and/or software that may combine image data of a captured live action scene and a rendered scene to form composited imagery.”).
Regarding claim 17, Thurston as modified by Stoski and Schmirler discloses the computer-implemented method of claim 16, wherein said providing augmented video data comprises compositing one or more video frames provided by a video camera and the diorama to provide the augmented video data (Thurston: para 51, “a compositor may correspond to hardware and/or software that may combine image data of a captured live action scene and a rendered scene to form composited imagery.”; also, para 17, “pixels representing the given virtual scene element in a composite image, captured by the camera, of the stage environment and the virtual scene display.”).
Regarding claim 18, Thurston as modified by Stoski and Schmirler discloses the computer-implemented method of claim 17, wherein the virtual camera is further set based on camera viewpoint data for the video camera to specify the perspective view of the diorama relative to the scaled waypoint scene model (Thurston: para 17, “the depth value corresponds to a virtual distance in the virtual scene between the given virtual scene element and a virtual camera viewpoint located at the camera position"; also, para 10, “determining the camera position includes reading data from camera position sensors placed on the camera.”).
Regarding claim 19, Thurston as modified by Stoski and Schmirler discloses the computer-implemented method of claim 18, output device is a portable device and is one of a mobile phone, a tablet, a television (TV) device, and a laptop computer.
However, in a similar field of endeavor, Stoski discloses wherein the output device is a portable device and is one of a mobile phone, a tablet, a television (TV) device, and a laptop computer (para 34, “A client device 310 is preferably a portable wireless device that can be carried by a player, such as a smartphone, portable gaming device, augmented reality (AR) headset, cellular phone, tablet, personal digital assistant (PDA), navigation system, handheld GPS system, or other such device.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of outputting a rendering on an output device with the features of Stoski's invention of a portable client device such as a smartphone or tablet. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Stoski expressly teaches that the client device on which the augmented content is presented "is preferably a portable wireless device that can be carried by a player, such as a smartphone, portable gaming device, augmented reality (AR) headset, cellular phone, tablet," which is a portable device that is one of a mobile phone and a tablet as recited. Second, presenting the augmented video on a portable client device lets a viewer experience the path-driven content of the scene away from a fixed display. Third, because Stoski already renders augmented content of the same general kind on such portable devices, a person of ordinary skill would have output the augmented video to a mobile phone or tablet with a predictable result.
Regarding claim 20, Thurston as modified by Stoski and Schmirler discloses the computer-implemented method of claim 19, diorama is animated to provide a visual representation of the scene from the perspective view.
However, in a similar field of endeavor, Schmirler discloses wherein the diorama is animated to provide a visual representation of the scene from the perspective view (para 100, “rendering component 308 renders a scaled-down version of the production area that can be viewed by the user as if the production area were an animated, interactive scale model.”; also, para 86, “the plant model can define physical dimensions and colors for the asset, as well as any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Thurston's invention of rendering a diorama from a virtual camera perspective view with the features of Schmirler's invention of presenting the scaled-down scene as an animated scale model. One of ordinary skill in the art would have been motivated to make this combination for the following reasons. First, Schmirler teaches that the rendering component "renders a scaled-down version of the production area that can be viewed by the user as if the production area were an animated, interactive scale model," which animates the diorama for the viewer. Second, Schmirler further teaches that the model can define "any animation supported by the graphical representation (e.g., color change animations, position animations that reflect movement of the asset, etc.)," which provides a visual representation of the scene that changes over time. Third, animating the diorama gives the viewer a continuous visual representation of the captured scene from the perspective view, improving comprehension of the spatial layout and the authored path, so a person of ordinary skill would have animated the diorama as taught by Schmirler with a predictable result.
Response to Arguments
Applicant's arguments filed April 29, 2026 have been fully considered but they are not persuasive.
In view of Applicant's amendments, the rejection of claims 1 to 6 and 8 to 10 under 35 U.S.C. 102(a)(2) over Thurston and the rejection of claims 7 and 11 to 20 under 35 U.S.C. 103 over Thurston in view of Major have been recast under the new grounds of rejection set forth above, which are necessitated by the amendments. Applicant's arguments are directed to Thurston considered alone and to Major, and do not address the teachings of Schmirler and Stoski now applied to the amended limitations.
Applicant argues at pages 28 to 31 of the Remarks that the amended claim 1 limitation of creating a miniaturized version of the scene model by scaling the scene model to provide a diorama distinguishes over Thurston because Thurston's virtual scene is a rendered scene presented on a display wall rather than a scaled miniature representation. This argument is not persuasive. The amended limitation is rejected above under 35 U.S.C. 103 over Thurston in view of Schmirler, not over Thurston alone. Schmirler teaches that "the system can render a scaled down view of the factory floor area, which affords the user an external overview of the area" and that "the external view generated by VR/AR presentation system 302 renders the industrial area as a virtual scale model of the area, and allows the user to move around and interact with the scaled version of the area," which is a miniaturized version of the scene produced by scaling, that is, a diorama. Applicant's argument addresses only what Thurston discloses and does not address Schmirler's teaching of rendering a captured area as a scaled-down three-dimensional scale model, and therefore does not identify error in the rejection.
Applicant argues at pages 31 to 33 of the Remarks that the hotspots of Major are interface controls rather than path-defining waypoints, and that neither Thurston nor Major teaches the amended waypoint-path features of claim 11. This argument is not persuasive because the present rejection does not rely on Major. The amended limitations of receiving waypoint instructions identifying virtual points that define a path for a digital asset, and of providing a waypoint scene model that stores the path for the digital asset, are rejected above under 35 U.S.C. 103 over Thurston in view of Stoski. Stoski teaches that "the user can define a path for a virtual object to follow," that "A route 405 is formed according to the defined waypoints," and that "the mapping system 327 stores the 3D map along with any semantic/contextual information in the 3D map store 328," on which "a waypoint is specified on a 3D map of the real world." Applicant's argument is directed to the hotspots of Major and does not address Stoski's teaching of waypoints that define and store a path for a virtual object, and therefore does not identify error in the rejection.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights