DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Office Action is in response to Applicant’s amendment filed 07/22/2026 which has
been entered and made of record. Claims 1, 11, 16, and 18 have been amended.
Claims 9, 12, and 17 have been cancelled. Claims 21-23 newly added. Claims 1-8, 10-11, 13-16, and 18-23 are pending in the application.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 11 and 16 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument (due to applicant’s arguments directed to newly amend limitation(s) which is addressed by new prior art presented in this Office Action).
The arguments regarding dependent claims for the virtue of their dependency are moot
because the independent claims are not allowable.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 11, 13, 16 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hornstein et al. (U.S. Patent Application Publication No. 2022/0303524), hereinafter referenced as Hornstein, in view of ROUTHIER (U.S. Patent Application Publication No. 2020/0177862), hereinafter referenced as Routhier.
Regarding claim 11, Hornstein teaches A method comprising: receiving image sensor data from a camera system on a computing device (fig. 4, and paragraph 45 teach “S100 can include acquiring the photoset (e.g., using an image acquisition system),…and using one or more virtual cameras to capture images of the render (e.g., to form a photoset)”); this shows image sensor data from camera being acquired on computing device/imaging acquisition system; the image sensor data representing a first portion of a physical environment, (paragraph 33 teaches “within the display region (e.g., cameras can be embedded proximal the light source), separate from the display (e.g., mounted in the same environment such as within a threshold distance of the viewer”); this shows since cameras (image sensor) mounted to same environment as viewer, the data thereof represents first portion of physical environment; transmitting input data to an image generation model, the input data including the image sensor data (paragraph 41 teaches “artificial intelligence module 230 can function to process one or more image of a photoset (e.g., to generate additional images or views for the photoset)…artificial intelligence module can include or use machine learning, neural network (e.g., convolutional neural network, recurrent neural network, etc.), Bayesian inference, and/or any suitable algorithms or machine learning”); this shows input of image sensor data (processed image(s) of photoset) transmitted to image generation model (neural network and machine learning in artificial intelligence model); receiving generative image content from the image generation model (paragraph 79 teaches “S400 can optionally include adjusting the lightfield image (and/or views or associated photoset) which functions to adjust a visual appearance of the lightfield image. The images are preferably adjusted based on a viewer and/or subject input, but can be adjust automatically (e.g., using a machine learning module to determine a preferred image setting)”); this shows adjusting image content using the machine learning/image generation model to generate and thus receive generative image content; the generative image content being based on a prediction of a second portion of the physical environment, the prediction being based on content of the image sensor data, (paragraph 45 teaches “image(s) of the photoset (e.g., for a given frame of a lightfield video) are preferably acquired contemporaneously (e.g., simultaneously, concurrently, within a threshold time, etc.), but can be acquired asynchronously. In some variations, receiving a photoset can include receiving (and/or generating) a render of a scene (e.g., a 3D render) and using one or more virtual cameras to capture images of the render (e.g., to form a photoset). S100 can optionally include generating one or more image (e.g., using artificial intelligence as shown for example in FIG. 9, by interpolating between images of the photoset, etc.) such as to include in (and/or form) the photoset.”); generative image content is shown by photoset which includes interpolated/predicted images (interpolated would be considered second portion of physical environment), and when viewed in combination with the above physical camera, this prediction/interpolation is based on content of aforementioned image sensor data; the second portion of the physical environment being peripheral to the field of view of the camera system (paragraph 36 teaches “The features can be specific structures (e.g., points, edges, corners...neighborhood size can depend on the feature size...When the neighborhood extends beyond an image boundary, the image can be filled in any manner (e.g., with extra pixels that are black, white, constant color, extrapolated data based on the image, extrapolated using artificial intelligence, interpolated from other images, etc.)"); feature being structure such as edges and/or corners which leads to neighborhood size determination which extends past boundary and is filled using interpolation shows that second portion of physical environment (interpolated) would be peripheral to field of view (FOV) of camera system since it is on edges or corners/peripheral of such; and generating display content by combining the image sensor data and the generative image content (fig. 4, steps S460-S480, paragraph 73 teaches “Overlapping the images S460 functions to overlap feature(s) of each image” and paragraph 76 teaches “S480 functions to combine the images (e.g., unprocessed images, cropped images, aligned images, adjusted images, processed images, etc.) to form a lightfield image”); this shows display content generated by combining unprocessed/image sensor data and adjusted images/generative image content as well as by overlapping features of each image; the generative image content being peripheral to the content based on the image sensor data and filling the field of view of the display of the computing device (paragraph 26 teaches this "the number of cameras in the image acquisition system can be less than the number of views (e.g., when one or more cameras are mounted on a gantry, track, robot, motor, and/or other movement system and acquire images from more than one perspective, when one or more intermediate views are interpolated or generated, etc.) or greater than the number of views "); this shows due to camera/image sensor having less view and interpolated fills the view of display, that the generated image content would be peripheral based on the image sensor data (to fill the missed views on the display where the camera can’t “see”).
However, Hornstein fails to teach a field of view of the camera system being smaller than a field of view of a display of the computing device;
However, Routhier teaches a field of view of the camera system being smaller than a field of view of a display of the computing device (Routhier, paragraph 86 teaches “field of view (FOV) of the display screen in FIG. 1C is larger than the original FOV of the camera images. As illustrated in FIG. 1C, qualitatively each image is placed within the single screen's window or frame with an appropriate sideways offset”); this shows FOV of camera smaller than FOV of display. Routhier is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of FOV comparisons for effective and aesthetic visual display. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Hornstein's invention with the FOV size techniques of Routhier to ensure the increase in quality of the 3D experience. This is due to taking into consideration the FOV when resizing.
Regarding claim 13, the combination of Hornstein and Routhier teaches wherein the input data includes a current image frame , one or more previous image frames, (Hornstein, fig. 4, step S200 teaches “for each image of the plurality of images” and paragraph 45 teaches “S100 can optionally include generating one or more image (e.g., using artificial intelligence as shown for example in FIG. 9, by interpolating between images of the photoset, etc.) such as to include in (and/or form) the photoset); interpolation shows previous and current image frames existing, also, since this is happening for each frame, once iterated to the next frame, that next frame becomes the current frame and there must be a previous frame; and pose information about an orientation of the computing device (Hornstein, paragraph 26 teaches “Each camera can be fixed (e.g., be mounted to have a static relation orientation, static absolute orientation, etc.) or moveable” and paragraph 42 teaches “characteristics of the system can include: camera pose (e.g., location, orientation, etc. for image acquisition system and/or each camera in the array)”); this shows input data must include the pose information about orientation of computing device/image acquisition system and camera thereof.
Regarding claim 16, A non-transitory computer-readable medium storing executable instructions that cause at least one processor to execute operations, the operations comprising: (paragraph 45 teaches “S100 can include acquiring the photoset (e.g., using an image acquisition system), retrieving the photoset (e.g., from a computing system such as from a database, storage, etc.), and/or include any suitable steps” and paragraph 34 teaches “control the image acquisition system and/or display, and/or perform any function(s)…computing system can be in communication with the image acquisition system, a subset of cameras of the image acquisition system, the display(s), and/or with any suitable components.”); storage here shows the non-transitory computer readable medium, performing any functions would mean it stores executable instructions and done by any component means the processor executes these operations; receiving image sensor data from a camera system on a computing device, (fig. 4, and paragraph 45 teach “S100 can include acquiring the photoset (e.g., using an image acquisition system),…and using one or more virtual cameras to capture images of the render (e.g., to form a photoset)”); this shows image sensor data from camera being acquired on computing device/imaging acquisition system; transmitting input data to an image generation model, the input data including the image sensor data (paragraph 41 teaches “artificial intelligence module 230 can function to process one or more image of a photoset (e.g., to generate additional images or views for the photoset)…artificial intelligence module can include or use machine learning, neural network (e.g., convolutional neural network, recurrent neural network, etc.), Bayesian inference, and/or any suitable algorithms or machine learning”); this shows input of image sensor data (processed image(s) of photoset) transmitted to image generation model (neural network and machine learning in artificial intelligence model); receiving generative image content from the image generation model(paragraph 79 teaches “S400 can optionally include adjusting the lightfield image (and/or views or associated photoset) which functions to adjust a visual appearance of the lightfield image. The images are preferably adjusted based on a viewer and/or subject input, but can be adjust automatically (e.g., using a machine learning module to determine a preferred image setting)”); this shows adjusting image content using the machine learning/image generation model to generate and thus receive generative image content; and generating display content for the display by combining the image sensor data and the generative image content, (fig. 4, steps S460-S480, paragraph 73 teaches “Overlapping the images S460 functions to overlap feature(s) of each image” and paragraph 76 teaches “S480 functions to combine the images (e.g., unprocessed images, cropped images, aligned images, adjusted images, processed images, etc.) to form a lightfield image”); this shows display content generated by combining unprocessed/image sensor data and adjusted images/generative image content as well as by overlapping features of each image;.
However, Hornstein fails to teach the camera system having a first field of view that is less than a second field of view of a display of the computing device; the generative image content expanding visual content based on the image sensor data from the first field of view to the second field of view;
However, Routhier teaches the camera system having a first field of view that is less than a second field of view of a display of the computing device (Routhier, paragraph 86 teaches “field of view (FOV) of the display screen in FIG. 1C is larger than the original FOV of the camera images. As illustrated in FIG. 1C, qualitatively each image is placed within the single screen's window or frame with an appropriate sideways offset”); this shows FOV of camera smaller than FOV of display; the generative image content expanding visual content based on the image sensor data from the first field of view to the second field of view (Routhier, paragraph 119 teaches “Esp>1, then the observer will perceive a wider world than reality (expanded)”); this shows generative image content expanding visual content and is based on image sensor data from first FOV to second FOV since happens in a step after listing the example of having display screen with larger FOV than original FOV of camera images. Routhier is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of FOV comparisons for effective and aesthetic visual display. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Hornstein's invention with the FOV size techniques of Routhier to ensure the increase in quality of the 3D experience. This is due to taking into consideration the FOV when resizing.
Regarding claim 19, the non-transitory computer-readable medium claim 19 recites similar limitations as method claim 13 and thus is rejected under similar rationale.
Claim(s) 1-2, 4-6, 8, 21, and 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hornstein in view of Routhier and HWANG (U.S. Patent Application Publication No. 2024/0303781), hereinafter referenced as Hwang.
Regarding claim 1, Hornstein teaches A computing device comprising: (paragraph 28 teaches “an image acquisition system can be a camera of a user device (e.g., smart phone)”; at least one processor (title, teaches “processing” which must be done by a processor); and a non-transitory computer readable medium storing executable instructions that cause the at least one processor to execute operations, (paragraph 45 teaches “S100 can include acquiring the photoset (e.g., using an image acquisition system), retrieving the photoset (e.g., from a computing system such as from a database, storage, etc.), and/or include any suitable steps” and paragraph 34 teaches “control the image acquisition system and/or display, and/or perform any function(s)…computing system can be in communication with the image acquisition system, a subset of cameras of the image acquisition system, the display(s), and/or with any suitable components.”); storage here shows the non-transitory computer readable medium, performing any functions would mean it stores executable instructions and done by any component means the processor executes these operations; the operations comprising: receiving image sensor data from a camera system on a computing device (fig. 4, and paragraph 45 teach “S100 can include acquiring the photoset (e.g., using an image acquisition system),…and using one or more virtual cameras to capture images of the render (e.g., to form a photoset)”); this shows image sensor data from camera being acquired on computing device/imaging acquisition system; transmitting input data to an image generation model, the input data including the image sensor data (paragraph 41 teaches “artificial intelligence module 230 can function to process one or more image of a photoset (e.g., to generate additional images or views for the photoset)…artificial intelligence module can include or use machine learning, neural network (e.g., convolutional neural network, recurrent neural network, etc.), Bayesian inference, and/or any suitable algorithms or machine learning”); this shows input of image sensor data (processed image(s) of photoset) transmitted to image generation model (neural network and machine learning in artificial intelligence model); receiving generative image content from the image generation model (paragraph 79 teaches “S400 can optionally include adjusting the lightfield image (and/or views or associated photoset) which functions to adjust a visual appearance of the lightfield image. The images are preferably adjusted based on a viewer and/or subject input, but can be adjust automatically (e.g., using a machine learning module to determine a preferred image setting)”); this shows adjusting image content using the machine learning/image generation model to generate and thus receive generative image content; and generating display content by combining the image sensor data and the generative image content (fig. 4, steps S460-S480, paragraph 73 teaches “Overlapping the images S460 functions to overlap feature(s) of each image” and paragraph 76 teaches “S480 functions to combine the images (e.g., unprocessed images, cropped images, aligned images, adjusted images, processed images, etc.) to form a lightfield image”); this shows display content generated by combining unprocessed/image sensor data and adjusted images/generative image content as well as by overlapping features of each image.
However, Hornstein fails to teach the image sensor data including pass-through video and the generative image content extending the pass-through video to a wider field of view;
However, Routhier teaches the image sensor data including video and the generative image content extending the video to a wider field of view (Routhier, paragraph 86 teaches “field of view (FOV) of the display screen in FIG. 1C is larger than the original FOV of the camera images. As illustrated in FIG. 1C, qualitatively each image is placed within the single screen's window or frame with an appropriate sideways offset” and paragraph 119 teaches “Esp>1, then the observer will perceive a wider world than reality (expanded)”); camera images as series would be video and the generative content would extend the video to wider FOV because of the expanding of perception mentioned and display having larger FOV than camera. Routhier is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of FOV comparisons for effective and aesthetic visual display. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Hornstein's invention with the FOV size techniques of Routhier to ensure the increase in quality of the 3D experience. This is due to taking into consideration the FOV when resizing.
However, the combination of Hornstein and Routhier fails to teach pass-through video.
However, Hwang teaches pass-through video (Hwang, paragraph 65 teaches “the display(s) of the output device(s) 235 of the sensor data processing system 200 are digital “pass-through” display that allow the user of the sensor data processing system 200 and/or a recipient device to see a view of an environment by displaying the view of the environment on the display(s) of the output device(s) 235. The view of the environment that is displayed on the digital pass-through display can be a view of the real-world environment around the sensor data processing system 200”). Hwang is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of having enhancing user experience by having pass-through display. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Hornstein and Routhier with the pass-through techniques of Hwang to provide a number of technical improvements over prior imaging systems, for instance processing images so that hues, tones, and or other characteristics of representations of certain types objects (e.g., skin, faces, hands, vegetation, sky, water, or other object types) are more accurate (Hwang, paragraph 37). This would be due to the pass-through technology passing the characteristics of the images directly to the video by preserving image characteristics.
Regarding claim 2, the combination of Hornstein, Routhier and Hwang teaches wherein the camera system has a first field of view that is less than a second field of view of a display of the computing device(Routhier, paragraph 86 teaches “field of view (FOV) of the display screen in FIG. 1C is larger than the original FOV of the camera images. As illustrated in FIG. 1C, qualitatively each image is placed within the single screen's window or frame with an appropriate sideways offset”); this shows FOV of camera smaller than FOV of display of computing device. The same motivations used in claim 1 apply here in claim 2.
Regarding claim 4, the combination of Hornstein, Routhier and Hwang teaches wherein the input data includes a current image frame (Hornstein, fig. 4, step S200 and paragraph 47 teaches “Determining a feature S200 functions to determine one or more feature in each image of the photoset”); this shows each image as input data (during iteration each frame being worked on is considered current image frame).
Regarding claim 5, the combination of Hornstein, Routhier and Hwang teaches wherein the input data includes a current image frame and one or more previous image frames (Hornstein, fig. 4, step S200 teaches “for each image of the plurality of images” and paragraph 45 teaches “S100 can optionally include generating one or more image (e.g., using artificial intelligence as shown for example in FIG. 9, by interpolating between images of the photoset, etc.) such as to include in (and/or form) the photoset); interpolation shows previous and current image frames existing, also, since this is happening for each frame, once iterated to the next frame, that next frame becomes the current frame and there must be a previous frame.
Regarding claim 6, the combination of Hornstein, Routhier and Hwang teaches wherein the input data includes pose information about an orientation of the computing device (Hornstein, paragraph 26 teaches “Each camera can be fixed (e.g., be mounted to have a static relation orientation, static absolute orientation, etc.) or moveable” and paragraph 42 teaches “characteristics of the system can include: camera pose (e.g., location, orientation, etc. for image acquisition system and/or each camera in the array)”); this shows input data must include the pose information about orientation of computing device/image acquisition system and camera thereof.
Regarding claim 8, the combination of Hornstein, Routhier and Hwang teaches wherein the operations further comprise: receiving camera parameter information about one or more camera parameters of the camera system (Hornstein, paragraph 27 teaches “a camera parameter (e.g., the camera frame rate, the camera resolution, the camera field of view, a stereo-camera baseline, frame rate, image resolution, etc.)” and paragraph 28 teaches “image acquisition system is preferably calibrated (e.g., camera pose for each camera known, intrinsic parameters for each camera known, extrinsic parameters for each camera known, etc.)”); this shows calibrating (thus must first receive) camera parameter information about the camera parameters of the camera system; and adjusting at least one of the image sensor data or the generative image content based on the camera parameter information (Hornstein, paragraph 37 teaches “alignment module 220 can function to align images of the photoset (e.g., generate an aligned photoset 115), modify images of the photoset (e.g., to change a focus of an image), correct for a variation in camera parameters, and/or can otherwise function”); this shows adjusting image sensor data and generative image content by correcting variation in camera parameters (thus based on camera parameter information).
Regarding claim 21, the combination of Hornstein, Routhier and Hwang teaches wherein the operations further comprise: determining an area of focus of a user based on data received from an eye gaze tracker (Hornstein, paragraph 74 teaches "The features to overlap (e.g., the zero-disparity location) can depend on a subject (e.g., subject activity such as to focus on an active subject; a duration since a subject was in focus; etc.), a viewer (e.g., a viewer preference, a viewer gaze such as to focus on a portion or feature of the image that the viewer is looking at as determined using a gaze tracker, etc.),"); viewer gaze such as to focus shows area of focus determined of user/viewer based on data received from eye gaze tracker; and rendering high-resolution details in the area of focus within the display content (Hwang, paragraph 46 teaches “merging of image frames to form an HDR image, image recognition, object recognition, feature recognition, receipt of inputs, managing outputs, managing memory, or some combination thereof”); this shows HDR image for object/feature recognition, the feature/object recognized would be the area of focus within display content and HDR for such shows rendering high resolution details in it. The same motivations used in claim 1 apply here in claim 21.
Regarding claim 23, the combination of Hornstein, Routhier and Hwang teaches wherein: the display content is first display content; the generative image content is first generative image content(Hornstein, fig. 4, steps S460-S480, paragraph 73 teaches “Overlapping the images S460 functions to overlap feature(s) of each image” and paragraph 76 teaches “S480 functions to combine the images (e.g., unprocessed images, cropped images, aligned images, adjusted images, processed images, etc.) to form a lightfield image”); this shows display content (which is considered first display content) generated by combining unprocessed/image sensor data and adjusted images/generative image content (which is considered first generative image content) as well as by overlapping features of each image; and the operations further comprise: generating second generative image content based on the first generative image content (Hornstein, paragraph 45 teaches “receiving a photoset can include receiving (and/or generating) a render of a scene (e.g., a 3D render) and using one or more virtual cameras to capture images of the render (e.g., to form a photoset). S100 can optionally include generating one or more image”); one or more image and generating render of scene shows second generative image content would exist and be based on the first since comes after as part of photoset images; and generating second display content by combining the image sensor data and the second generative image content (Hornstein, paragraph 45 teaches “can optionally include generating one or more image (e.g., using artificial intelligence as shown for example in FIG. 9, by interpolating between images of the photoset, etc.) such as to include in (and/or form) the photoset”); one or more images generated from interpolation of other photoset images shows second display content generated by combining image sensor data and the aforementioned second generative image content when those two images form the photoset.
Claim(s) 14 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hornstein and Routhier as applied to claims 11 and 16 above, and further in view of Rudman et al. (U.S. Patent No. 11,948,263), hereinafter referenced as Rudman.
Regarding claim 14, the combination of Hornstein and Routhier fails to teach further comprising: applying a transparency blend to the image sensor data and the generative image content.
However, Rudman teaches further comprising: applying a transparency blend to the image sensor data and the generative image content (Rudman, col. 71, lines 55 and 57-60 teach “virtual object may be displayed…at least partially transparently (e.g., allowing the user to at least partially see a physical environment through the virtual object)” and col. 41, lines 31-38 teach ” User 602 may view information 604 displayed on electronic screen 610 through a transparent section of wearable extended reality appliance 608 (e.g., where pixels of wearable extended reality appliance 608 are inactivate) while concurrently viewing virtual content in a non-transparent section of wearable extended reality appliance 608 (e.g., including activated pixels of wearable extended reality appliance 608).”); virtual objects being transparent shows transparency blend to generative image content and electronic screen 610 (as aforementioned in claim 2, captured from camera 626) viewed through a transparent section also shows transparency blend of image sensor data. Rudman is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of field of view of camera and display devices in XR and headset technology. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Hornstein and Routhier with the FOV techniques of Rudman to provide a mobile environment that enables users to experience the comfort of a stationary workspace anywhere they want by providing virtual desktop-like screens. (Rudman, col. 1, lines 51-55). This ensures a more convenient and better user experience by increasing mobility and due to the way the virtual screen field of view (FOV) is configured.
Regarding claim 18, the combination of Hornstein, Routhier and Rudman teaches wherein the generative image content includes a peripheral portion between the first field of view and the second field of view (Rudman, col. 44, lines 38-43 teach “FIG. 6A illustrates a control button 622 (e.g., “GOTO XR”) for altering a location of user interface 612. Pressing control button 622 may cause user interface 612 to be displayed in peripheral region 618, e.g., external to the boundaries of electronic screen 610, to alter the location of user interface 612”); this alongside fig. 6B shows generative image content/user interface 612 being displayed in peripheral region 618 which is between the first field of view/boundaries of camera field of view 610 and the second field of view/boundaries of display [of wearable extended reality appliance 608] of device 618 which is described as a peripheral region/portion. The same motivations used in claim 2 apply here in claim 3.
Claim(s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hornstein and Routhier as applied to claim 16 above, and further in view of Bastani et al. (U.S. Patent No. 11,127,221), hereinafter referenced as Bastani.
Regarding claim 20, the combination of Hornstein and Routhier teaches wherein the operations further comprise: generating a first image view with the image sensor data and the generative image content (paragraph 59 teaches “Processing the photoset S400 functions to process images of the photoset, to generate views associated with a lightfield image” and paragraph 76 teaches “S480 functions to combine the images (e.g., unprocessed images, cropped images, aligned images, adjusted images, processed images, etc.) to form a lightfield image” ); this shows lightfield image generated using the image sensor data/unprocessed images and generative image content/adjusted images, and then a view associated with the lightfield image, thus, this shows a first image view with image sensor data and generative image content (since the lightfield image view has both).
However, the combination of Hornstein and Routhier fails to explicitly teach and generating a second image view by re-projecting the first image view.
However, Bastani teaches and generating a second image view by re-projecting the first image view (Bastani, col. 8, lines 64-67 teach “the reprojection may be performed on the image frame to reuse a portion of the image frame to generate the updated image frame of a view of the artificial reality corresponding to the updated sensor measurements”); updated image frame of a view shows a second image view generated and this is done by reprojecting the first image view. Bastani is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of reprojecting image views. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Hornstein and Routhier with the reprojecting techniques of Bastani so communication bandwidth between the console 110 and the HWD 150 can be reduced, and a high resolution image can be presented to the user without sacrificing fidelity (Bastani, col. 9, lines 1-4). This would be done by reusing previously rendered information by reprojection thus reducing the load to render completely new frames every time.
Claim(s) 3, and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hornstein, Routhier and Hwang as applied to claim 1-2 above, and further in view of Rudman.
Regarding claim 3, the combination of Hornstein, Routhier and Hwang fails to explicitly teach wherein the generative image content includes a peripheral portion between the first field of view and the second field of view.
However, Rudman teaches wherein the generative image content includes a peripheral portion between the first field of view and the second field of view (Rudman, col. 44, lines 38-43 teach “FIG. 6A illustrates a control button 622 (e.g., “GOTO XR”) for altering a location of user interface 612. Pressing control button 622 may cause user interface 612 to be displayed in peripheral region 618, e.g., external to the boundaries of electronic screen 610, to alter the location of user interface 612”); this alongside fig. 6B shows generative image content/user interface 612 being displayed in peripheral region 618 which is between the first field of view/boundaries of camera field of view 610 and the second field of view/boundaries of display [of wearable extended reality appliance 608] of device 618 which is described as a peripheral region/portion. Rudman is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of field of view of camera and display devices in XR and headset technology. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Hornstein's invention with the FOV techniques of Rudman to provide a mobile environment that enables users to experience the comfort of a stationary workspace anywhere they want by providing virtual desktop-like screens. (Rudman, col. 1, lines 51-55). This ensures a more convenient and better user experience by increasing mobility and due to the way the virtual screen field of view (FOV) is configured.
Regarding claim 7, the combination of Hornstein, Routhier, Hwang and Rudman teaches wherein the operations further comprise: applying a transparency blend to the image sensor data and the generative image content (Rudman, col. 71, lines 55 and 57-60 teach “virtual object may be displayed…at least partially transparently (e.g., allowing the user to at least partially see a physical environment through the virtual object)” and col. 41, lines 31-38 teach ” User 602 may view information 604 displayed on electronic screen 610 through a transparent section of wearable extended reality appliance 608 (e.g., where pixels of wearable extended reality appliance 608 are inactivate) while concurrently viewing virtual content in a non-transparent section of wearable extended reality appliance 608 (e.g., including activated pixels of wearable extended reality appliance 608).”); virtual objects being transparent shows transparency blend to generative image content and electronic screen 610 (as aforementioned in claim 2, captured from camera 626) viewed through a transparent section also shows transparency blend of image sensor data. The same motivations used in claim 3 apply here in claim 7.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hornstein, Routhier and Hwang as applied to claim 1 above, and further in view of Ha et al. (U.S. Patent Application Publication No. 2021/0201586), hereinafter referenced as Ha.
Regarding claim 10, the combination of Hornstein, Routhier and Hwang fails to explicitly teach wherein the operations further comprise: transmitting, over a network, the input data to the image generation model; and receiving, over the network, generative image sensor data from the image generation model.
However, Ha teaches wherein the operations further comprise: transmitting, over a network, the input data to the image generation model (Ha, paragraph 232 teaches “an image transmitted or received over a network”); this shows image (such as captured image/input data from Hornstein) transmitted over a network and when viewed in combination and as explained in claim 1 above this transmitting is to the image generation model from Hornstein; and receiving, over the network, generative image sensor data from the image generation model (Ha, paragraph 236 teaches “generating an image converted from a given area of the image of the 3D space transmitted or received over a network”); this shows receiving a generated image/generative image sensor data (which would be from image generation model when viewed in combination and as explained in claim 1 above) over the network. Ha is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of transmitting and receiving image data over a network. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Hornstein, Routhier and Hwang with the network techniques of Ha to ensure providing a security function for a three-dimensional (3D) space and, more particularly, to an AR remote communication method and system (Ha, paragraph 2). This would be done using the network as a secure means of transmittance and reception of image data leading to a more secure system overall.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hornstein, Routhier and Hwang as applied to claim 1 above, and further in view of Bastani.
Regarding claim 15, the combination of Hornstein, Routhier and Hwang teaches further comprising: generating a first image view with the image sensor data and the generative image content (paragraph 59 teaches “Processing the photoset S400 functions to process images of the photoset, to generate views associated with a lightfield image” and paragraph 76 teaches “S480 functions to combine the images (e.g., unprocessed images, cropped images, aligned images, adjusted images, processed images, etc.) to form a lightfield image” ); this shows lightfield image generated using the image sensor data/unprocessed images and generative image content/adjusted images, and then a view associated with the lightfield image, thus, this shows a first image view with image sensor data and generative image content (since the lightfield image view has both).
However, the combination of Hornstein, Routhier and Hwang fails to explicitly teach and generating a second image view by re-projecting the first image view.
However, Bastani teaches and generating a second image view by re-projecting the first image view (Bastani, col. 8, lines 64-67 teach “the reprojection may be performed on the image frame to reuse a portion of the image frame to generate the updated image frame of a view of the artificial reality corresponding to the updated sensor measurements”); updated image frame of a view shows a second image view generated and this is done by reprojecting the first image view. Bastani is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of reprojecting image views. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Hornstein, Routhier and Hwang with the reprojecting techniques of Bastani so communication bandwidth between the console 110 and the HWD 150 can be reduced, and a high resolution image can be presented to the user without sacrificing fidelity (Bastani, col. 9, lines 1-4). This would be done by reusing previously rendered information by reprojection thus reducing the load to render completely new frames every time.
Claim(s) 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Hornstein, Routhier and Hwang as applied to claim 1 above, and further in view of Hillman (U.S. Patent Application Publication No. 2022/0076437), hereinafter referenced as Hillman.
Regarding claim 22, the combination of Hornstein, Routhier and Hwang fails to teach wherein the operations further comprise adjusting a value of a pixel at an intersection of the image sensor data and the generative image content so that the value of the pixel at the intersection is between a value of a pixel of the image sensor data and a value of a pixel of the generative image content.
However, Hillman teaches wherein the operations further comprise adjusting a value of a pixel at an intersection of the image sensor data and the generative image content so that the value of the pixel at the intersection is between a value of a pixel of the image sensor data and a value of a pixel of the generative image content (Hillman, paragraph 5 teaches “image generator, such as a renderer, might compute pixel color values based on which objects are intersected by rays from the camera position through pixel areas of the camera viewing frame. In an example, a scene description might specify the positions, shapes, texture, colors, etc. of various objects and a renderer could generate an image of what the scene would look like from a specified camera position through a specified camera viewing frame” and abstract teaches “a modified pixel color value is determined from scaled blend color values”); this shows pixel color value computed at intersection between camera position (image sensor data) and pixel area of viewing frame (generative image content) and is done so that the pixel value can be modified to be a blend between the two as mentioned in abstract meaning that the value of the pixel is adjusted to be between a value of pixel in image sensor data and value of pixel of in the generative image content. Hillman is considered to be analogous art because it is reasonably pertinent to the problem faced by the inventor of modifying/adjusting pixel values to be between other pixel values. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Hornstein, Routhier and Hwang with the pixel blend techniques of Hillman to allow emulating a depth of field effect of a physical camera. (Hillman, paragraph 8). This would improve the quality of the rendering.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yu et al. (U.S. Patent No. 10,084,959), col. 4, lines 8-11 teach “a panoramic image may be created by one or more cameras that rotate to capture an image that displays a field of view that is larger than the field of view of the camera when stationary.”; this shows larger/wider FOV of display would be needed to properly display the panoramic image take from smaller FOV camera.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NAUMAN U AHMAD whose telephone number is (703)756-5306. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.U.A./ Examiner, Art Unit 2611
/KEE M TUNG/ Supervisory Patent Examiner, Art Unit 2611
