Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 3-14 and 19 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
35 USC § 101
Claims 1- 20 are considered to be patent eligible under 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 16, 17, 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Flaks et al. (US 20120092328 A1) as cited in an 892 in view of Boorboor et al., VoxAR: Adaptive Visualization of Volume Rendered Objects in Optical See-Through Augmented Reality, as cited in an 892.
Regarding claim 1, Flaks teaches a device comprising (See abstract, “A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content.”):
at least one camera capturing pass-through video of a physical environment (See ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real-world object.” ¶44, “Opacity filter 114 can include a dense grid of pixels, where the light transmissivity of each pixel is individually controllable between minimum and maximum transmissivities. While a transmissivity range of 0-100% is ideal, more limited ranges are also acceptable. As an example, a monochrome LCD panel with no more than two polarizing filters is sufficient to provide an opacity range of about 50% to 90% per pixel, up to the resolution of the LCD. At the minimum of 50%, the lens will have a slightly tinted appearance, which is tolerable. 100% transmissivity represents a perfectly clear lens. An "alpha" scale can be defined from 0-100%, where 0% allows no light to pass and 100% allows all light to pass. The value of alpha can be set for each pixel by the opacity filter control circuit 224 described below.” Said another way, the opacity value of the display can make the video a pass-through video. Also see ¶43.);
a display (See abstract, “A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content.”);
blend circuitry (See ¶39, “A portion of the frame of head mounted display device 2 will surround a display (that includes one or more lenses). In order to show the components of head mounted display device 2, a portion of the frame surrounding the display is not depicted. The display includes a light guide optical element 112, opacity filter 114, see-through lens 116 and see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, lightguide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with lightguide optical element 112. See-through lenses 116 and 118 are standard lenses used in eye glasses and can be made to any prescription (including no prescription). In one embodiment, see-through lenses 116 and 118 can be replaced by a variable prescription lens. In some embodiments, head mounted display device 2 will include only one see-through lens or no see-through lenses. In another alternative, a prescription lens can go inside lightguide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the virtual imagery. Lightguide optical element 112 channels artificial light to the eye. More detailed of opacity filter 114 and lightguide optical element 112 is provided below.” The blended circuitry is considered to be the opacity filter, since it can change the opacity values, as the value is changed in ¶45, it functions as blending.) that generates augmented reality video for the display from the pass- through video (See ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real-world object.” Fusing virtual content with real content is augmented reality.);
at least one processor (See abstract, “A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content.”); and
a non-transitory computer-readable storage medium comprising instructions that, when executed on the at least one processor, cause the device to perform operations comprising (See ¶.33, “Hub computing system 12 may be a computer, a gaming system or console, or the like. According to an example embodiment, the hub computing system 12 may include hardware components and/or software components such that hub computing system 12 may be used to execute applications such as gaming applications, non-gaming applications, or the like. In one embodiment, hub computing system 12 may include a processor such as a standardized processor, a specialized processor, a microprocessor, or the like that may execute instructions stored on a processor readable storage device for performing the processes described herein.” See Fig. 3 and ¶57, discloses the same components as ¶33, and thus is considered to be capable of doing the same thing as well.):
producing virtual content (See abstract, “A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content. In one embodiment, the processing unit is in communication with a hub computing device. The system creates a volumetric model of a space, segments the model into objects, identifies one or more of the objects including a first object, and displays a virtual image over the first object on a display (of the head mounted display) that allows actual direct viewing of at least a portion of the space through the display.”);
[…] and
controlling the blend circuitry (See ¶39, “A portion of the frame of head mounted display device 2 will surround a display (that includes one or more lenses). In order to show the components of head mounted display device 2, a portion of the frame surrounding the display is not depicted. The display includes a light guide optical element 112, opacity filter 114, see-through lens 116 and see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, lightguide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with lightguide optical element 112. See-through lenses 116 and 118 are standard lenses used in eye glasses and can be made to any prescription (including no prescription). In one embodiment, see-through lenses 116 and 118 can be replaced by a variable prescription lens. In some embodiments, head mounted display device 2 will include only one see-through lens or no see-through lenses. In another alternative, a prescription lens can go inside lightguide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the virtual imagery. Lightguide optical element 112 channels artificial light to the eye. More detailed of opacity filter 114 and lightguide optical element 112 is provided below.” The blended circuitry is considered to be the opacity filter, since it can change the opacity values, as the value is changed in ¶45, it functions as blending.) to generate the augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass-through video […] from the virtual content (Generates augmented reality video for the display from the pass- through video see ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real world object.” Fusing virtual content with real content is augmented reality.) but doesn’t explicit disclose: calculating a color from the virtual content; with the calculated color
Boorboor teaches calculating a color from the virtual content (See abstract, “We present VoxAR, a method to facilitate an effective visualization of volume-rendered objects in optical see through head-mounted displays (OST-HMDs). The potential of augmented reality (AR) to integrate digital information into the physical world provides new opportunities for visualizing and interpreting scientific data. However, a limitation of OSTHMD technology is that rendered pixels of a virtual object can interfere with the colors of the real-world, making it challenging to perceive the augmented virtual information accurately. We address this challenge in a two-step approach. First, VoxAR
determines an appropriate placement of the volume-rendered object in the real-world scene by evaluating a set of spatial and environmental objectives, managed as user-selected preferences and pre-defined constraints. We achieve a real-time solution
by implementing the objectives using a GPU shader language. Next, VoxAR adjusts the colors of the input transfer function (TF) based on the real-world placement region. Specifically, we introduce a novel optimization method that adjusts the TF colors such that the resulting volume-rendered pixels are discernible against the background and the TF maintains the perceptual mapping between the colors and data intensity values. Finally, we present an assessment of our approach through objective evaluations and subjective user studies.”); […] with the calculated color […] (See abstract and previous quotes.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Flaks in view of Boorboor as Boorboor “adjusts the TF colors such that the resulting volume-rendered pixels are discernible against the background and the TF maintains the perceptual mapping between the colors and data intensity values.“ (Boorboor) and offers significant advantages primarily related to enhanced realism, immersion, and user experience.
Regarding claim 2, Flaks in view of Boorboor teaches the device of claim 1, wherein the blend circuitry comprises a dedicated pathway to the at least one camera (See Flaks ¶38-39. ¶38,” FIG. 2 depicts a top view of a portion of head mounted display device 2, including a portion of the frame that includes temple 102 and nose bridge 104. Only the right side of head mounted display device 2 is depicted. Built into nose bridge 104 is a microphone 110 for recording sounds and transmitting that audio data to processing unit 4, as described below. At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶39, “A portion of the frame of head mounted display device 2 will surround a display (that includes one or more lenses). In order to show the components of head mounted display device 2, a portion of the frame surrounding the display is not depicted. The display includes a light guide optical element 112, opacity filter 114, see-through lens 116 and see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, lightguide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with lightguide optical element 112. See-through lenses 116 and 118 are standard lenses used in eye glasses and can be made to any prescription (including no prescription). In one embodiment, see-through lenses 116 and 118 can be replaced by a variable prescription lens. In some embodiments, head mounted display device 2 will include only one see-through lens or no see-through lenses. In another alternative, a prescription lens can go inside lightguide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the virtual imagery. Lightguide optical element 112 channels artificial light to the eye. More detailed of opacity filter 114 and lightguide optical element 112 is provided below.” See Fig. 3, ¶56-¶57).
PNG
media_image1.png
820
1042
media_image1.png
Greyscale
Figure 3, with annotation clarifying the connections between the Opacity Filter (blending circuit) and the cameras.
Regarding claim 16, Flaks in view of Boorboor teaches the device of claim 1, wherein the operations further comprise displaying the augmented reality video (See Flaks abstract, “A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content.” Fusing virtual content into real content is describing augmented reality. ¶86, “A graphics processing unit (GPU) 508 and a video encoder/video codec (coder/decoder) 514 form a video processing pipeline for high speed and high resolution graphics processing. Data is carried from the graphics processing unit 508 to the video encoder/video codec 514 via a bus. The video processing pipeline outputs data to an A/V (audio/video) port 540 for transmission to a television or other display. A memory controller 510 is connected to the GPU 508 to facilitate processor access to various types of memory 512, such as, but not limited to, a RAM (Random Access Memory).”).
Regarding claim 17, Flaks teaches a method (See claim 1, “A method for fusing virtual content into real content […]”) comprising:
at a device having, at least one processor (See abstract, “A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content.”), at least one camera, a display (See ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real world object.” ¶44, “Opacity filter 114 can include a dense grid of pixels, where the light transmissivity of each pixel is individually controllable between minimum and maximum transmissivities. While a transmissivity range of 0-100% is ideal, more limited ranges are also acceptable. As an example, a monochrome LCD panel with no more than two polarizing filters is sufficient to provide an opacity range of about 50% to 90% per pixel, up to the resolution of the LCD. At the minimum of 50%, the lens will have a slightly tinted appearance, which is tolerable. 100% transmissivity represents a perfectly clear lens. An "alpha" scale can be defined from 0-100%, where 0% allows no light to pass and 100% allows all light to pass. The value of alpha can be set for each pixel by the opacity filter control circuit 224 described below.”), and blend circuitry (See ¶39, “A portion of the frame of head mounted display device 2 will surround a display (that includes one or more lenses). In order to show the components of head mounted display device 2, a portion of the frame surrounding the display is not depicted. The display includes a light guide optical element 112, opacity filter 114, see-through lens 116 and see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, lightguide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with lightguide optical element 112. See-through lenses 116 and 118 are standard lenses used in eye glasses and can be made to any prescription (including no prescription). In one embodiment, see-through lenses 116 and 118 can be replaced by a variable prescription lens. In some embodiments, head mounted display device 2 will include only one see-through lens or no see-through lenses. In another alternative, a prescription lens can go inside lightguide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the virtual imagery. Lightguide optical element 112 channels artificial light to the eye. More detailed of opacity filter 114 and lightguide optical element 112 is provided below.” The blended circuitry is considered to be the opacity filter, since it can change the opacity values, as the value is changed in ¶45, it functions as blending.):
capturing pass-through video of a physical environment via the at least one camera (See ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real world object.” ¶44, “Opacity filter 114 can include a dense grid of pixels, where the light transmissivity of each pixel is individually controllable between minimum and maximum transmissivities. While a transmissivity range of 0-100% is ideal, more limited ranges are also acceptable. As an example, a monochrome LCD panel with no more than two polarizing filters is sufficient to provide an opacity range of about 50% to 90% per pixel, up to the resolution of the LCD. At the minimum of 50%, the lens will have a slightly tinted appearance, which is tolerable. 100% transmissivity represents a perfectly clear lens. An "alpha" scale can be defined from 0-100%, where 0% allows no light to pass and 100% allows all light to pass. The value of alpha can be set for each pixel by the opacity filter control circuit 224 described below.” Said another way, the opacity value of the display can make the video a pass-through video. Also see ¶43.);
producing virtual content (See abstract, “A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content. In one embodiment, the processing unit is in communication with a hub computing device. The system creates a volumetric model of a space, segments the model into objects, identifies one or more of the objects including a first object, and displays a virtual image over the first object on a display (of the head mounted display) that allows actual direct viewing of at least a portion of the space through the display.”);
[…]; and
controlling the blend circuitry (See ¶39, “A portion of the frame of head mounted display device 2 will surround a display (that includes one or more lenses). In order to show the components of head mounted display device 2, a portion of the frame surrounding the display is not depicted. The display includes a light guide optical element 112, opacity filter 114, see-through lens 116 and see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, lightguide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with lightguide optical element 112. See-through lenses 116 and 118 are standard lenses used in eye glasses and can be made to any prescription (including no prescription). In one embodiment, see-through lenses 116 and 118 can be replaced by a variable prescription lens. In some embodiments, head mounted display device 2 will include only one see-through lens or no see-through lenses. In another alternative, a prescription lens can go inside lightguide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the virtual imagery. Lightguide optical element 112 channels artificial light to the eye. More detailed of opacity filter 114 and lightguide optical element 112 is provided below.” The blended circuitry is considered to be the opacity filter, since it can change the opacity values, as the value is changed in ¶45, it functions as blending.) to generate an augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass-through video […] from the virtual content (Generates augmented reality video for the display from the pass- through video see ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real world object.” Fusing virtual content with real content is augmented reality.) but doesn’t explicitly disclose calculating a color from the virtual content; with the calculated color.
Boorboor teaches calculating a color from the virtual content (See abstract, “We present VoxAR, a method to facilitate an effective visualization of volume-rendered objects in optical see through head-mounted displays (OST-HMDs). The potential of augmented reality (AR) to integrate digital information into the physical world provides new opportunities for visualizing and interpreting scientific data. However, a limitation of OSTHMD technology is that rendered pixels of a virtual object can interfere with the colors of the real-world, making it challenging to perceive the augmented virtual information accurately. We address this challenge in a two-step approach. First, VoxAR
determines an appropriate placement of the volume-rendered object in the real-world scene by evaluating a set of spatial and environmental objectives, managed as user-selected preferences and pre-defined constraints. We achieve a real-time solution
by implementing the objectives using a GPU shader language. Next, VoxAR adjusts the colors of the input transfer function (TF) based on the real-world placement region. Specifically, we introduce a novel optimization method that adjusts the TF colors such that the resulting volume-rendered pixels are discernible against the background and the TF maintains the perceptual mapping between the colors and data intensity values. Finally, we present an assessment of our approach through objective evaluations and subjective user studies.”); […] with the calculated color […] (See abstract and previous quotes.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Flaks in view of Boorboor as Boorboor “adjusts the TF colors such that the resulting volume-rendered pixels are discernible against the background and the TF maintains the perceptual mapping between the colors and data intensity values. “(Boorboor) and offers significant advantages primarily related to enhanced realism, immersion, and user experience.
Regarding claim 18, Flaks in view of Boorboor teaches the method of claim 17, wherein the blend circuitry comprises a dedicated pathway to the at least one camera (See Flaks ¶38-39. ¶38,” FIG. 2 depicts a top view of a portion of head mounted display device 2, including a portion of the frame that includes temple 102 and nose bridge 104. Only the right side of head mounted display device 2 is depicted. Built into nose bridge 104 is a microphone 110 for recording sounds and transmitting that audio data to processing unit 4, as described below. At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶39, “A portion of the frame of head mounted display device 2 will surround a display (that includes one or more lenses). In order to show the components of head mounted display device 2, a portion of the frame surrounding the display is not depicted. The display includes a light guide optical element 112, opacity filter 114, see-through lens 116 and see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, lightguide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with lightguide optical element 112. See-through lenses 116 and 118 are standard lenses used in eye glasses and can be made to any prescription (including no prescription). In one embodiment, see-through lenses 116 and 118 can be replaced by a variable prescription lens. In some embodiments, head mounted display device 2 will include only one see-through lens or no see-through lenses. In another alternative, a prescription lens can go inside lightguide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the virtual imagery. Lightguide optical element 112 channels artificial light to the eye. More detailed of opacity filter 114 and lightguide optical element 112 is provided below.” See Fig. 3, ¶56-¶57).
Regarding claim 20, Flaks teaches a non-transitory computer-readable storage medium storing program instructions executable via one or more processors (See ¶.33, “Hub computing system 12 may be a computer, a gaming system or console, or the like. According to an example embodiment, the hub computing system 12 may include hardware components and/or software components such that hub computing system 12 may be used to execute applications such as gaming applications, non-gaming applications, or the like. In one embodiment, hub computing system 12 may include a processor such as a standardized processor, a specialized processor, a microprocessor, or the like that may execute instructions stored on a processor readable storage device for performing the processes described herein.” See Fig. 3 and ¶57, discloses the same components as ¶33, and thus is considered to be capable of doing the same thing as well.), of a device having at least one camera, a display, and blend circuitry (See Fig. 3, ¶56-¶57), to perform operations comprising (See ¶.33, “Hub computing system 12 may be a computer, a gaming system or console, or the like. According to an example embodiment, the hub computing system 12 may include hardware components and/or software components such that hub computing system 12 may be used to execute applications such as gaming applications, non-gaming applications, or the like. In one embodiment, hub computing system 12 may include a processor such as a standardized processor, a specialized processor, a microprocessor, or the like that may execute instructions stored on a processor readable storage device for performing the processes described herein.” See Fig. 3 and ¶57, discloses the same components as ¶33, and thus is considered to be capable of doing the same thing as well.):
capturing pass-through video of a physical environment via the at least one camera;
producing virtual content (See ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real world object.” ¶44, “Opacity filter 114 can include a dense grid of pixels, where the light transmissivity of each pixel is individually controllable between minimum and maximum transmissivities. While a transmissivity range of 0-100% is ideal, more limited ranges are also acceptable. As an example, a monochrome LCD panel with no more than two polarizing filters is sufficient to provide an opacity range of about 50% to 90% per pixel, up to the resolution of the LCD. At the minimum of 50%, the lens will have a slightly tinted appearance, which is tolerable. 100% transmissivity represents a perfectly clear lens. An "alpha" scale can be defined from 0-100%, where 0% allows no light to pass and 100% allows all light to pass. The value of alpha can be set for each pixel by the opacity filter control circuit 224 described below.” Said another way, the opacity value of the display can make the video a pass-through video. Also see ¶43.);
[…]; and
controlling the blend circuitry (See ¶39, “A portion of the frame of head mounted display device 2 will surround a display (that includes one or more lenses). In order to show the components of head mounted display device 2, a portion of the frame surrounding the display is not depicted. The display includes a light guide optical element 112, opacity filter 114, see-through lens 116 and see-through lens 118. In one embodiment, opacity filter 114 is behind and aligned with see-through lens 116, lightguide optical element 112 is behind and aligned with opacity filter 114, and see-through lens 118 is behind and aligned with lightguide optical element 112. See-through lenses 116 and 118 are standard lenses used in eye glasses and can be made to any prescription (including no prescription). In one embodiment, see-through lenses 116 and 118 can be replaced by a variable prescription lens. In some embodiments, head mounted display device 2 will include only one see-through lens or no see-through lenses. In another alternative, a prescription lens can go inside lightguide optical element 112. Opacity filter 114 filters out natural light (either on a per pixel basis or uniformly) to enhance the contrast of the virtual imagery. Lightguide optical element 112 channels artificial light to the eye. More detailed of opacity filter 114 and lightguide optical element 112 is provided below.” The blended circuitry is considered to be the opacity filter, since it can change the opacity values, as the value is changed in ¶45, it functions as blending.) to generate an augmented reality video by blending the pass-through video with the virtual content and by modifying at least one portion of the pass- through video [… from the virtual content but doesn’t explicitly disclose calculating a color from the virtual content (Generates augmented reality video for the display from the pass- through video see ¶38, “At the front of head mounted display device 2 is room facing video camera 113 that can capture video and still images. Those images are transmitted to processing unit 4, as described below.” ¶4, “The technology described herein provides a system for fusing virtual content with real content such that an image of a virtual object may block all or part of the view of the real object. One or more sensors are used to scan the environment and build a model of the scanned environment. Using the model, the system adds a virtual image to the user's view of the environment at a location that is in reference to a real world object.” Fusing virtual content with real content is augmented reality.); with the calculated color.
Boorboor teaches calculating a color from the virtual content (See abstract, “We present VoxAR, a method to facilitate an effective visualization of volume-rendered objects in optical see through head-mounted displays (OST-HMDs). The potential of augmented reality (AR) to integrate digital information into the physical world provides new opportunities for visualizing and interpreting scientific data. However, a limitation of OSTHMD technology is that rendered pixels of a virtual object can interfere with the colors of the real-world, making it challenging to perceive the augmented virtual information accurately. We address this challenge in a two-step approach. First, VoxAR
determines an appropriate placement of the volume-rendered object in the real-world scene by evaluating a set of spatial and environmental objectives, managed as user-selected preferences and pre-defined constraints. We achieve a real-time solution
by implementing the objectives using a GPU shader language. Next, VoxAR adjusts the colors of the input transfer function (TF) based on the real-world placement region. Specifically, we introduce a novel optimization method that adjusts the TF colors such that the resulting volume-rendered pixels are discernible against the background and the TF maintains the perceptual mapping between the colors and data intensity values. Finally, we present an assessment of our approach through objective evaluations and subjective user studies.”); […] with the calculated color […] (See abstract and previous quotes.)
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Flaks in view of Boorboor as Boorboor “adjusts the TF colors such that the resulting volume-rendered pixels are discernible against the background and the TF maintains the perceptual mapping between the colors and data intensity values. “(Boorboor) and offers significant advantages primarily related to enhanced realism, immersion, and user experience.
Claims 15 is rejected under 35 U.S.C. 103 as being unpatentable over Flaks et al. (US 20120092328 A1) as cited in an 892 in view of Boorboor et al., VoxAR: Adaptive Visualization of Volume Rendered Objects in Optical See-Through Augmented Reality, as cited in an 892. Xiang et al. (US 20250245789 A1), as cited in an 892, filed, Jan. 29, 2024.
Regarding claim 15, Flaks in view of Boorboor teaches the device of claim 1, but doesn’t explicitly disclose wherein the augmented reality video has a frame rate greater than 60fps.
Xiang teaches wherein the augmented reality video has a frame rate greater than 60fps (See ¶3, “In certain applications, it may be desirable to increase the number of frames in a given video so that it can achieve a higher playback speed, such as 60 fps or 120 fps. For example, a display may be configured to output content at 60 fps, but generating a video at that same rate may be impractical or impossible due to system limitations (e.g., the computation resources may be limited, such as on a mobile device or virtual reality or augmented reality headset). Rendering a video at a high frame rate may also be challenging if the scene is complex and/or the desired resolution is high. In such cases, video interpolation may be used to convert a video with a relatively low frame rate (e.g., 30 fps) into a video with a higher frame rate (e.g., 60, 90, or 120 fps). As another example, some applications may wish to generate a slow-motion effect for a given video. If the video originally had 30 fps, doubling the number of frames by generating a synthesized frame between each pair of the original frames would result in 60 frames for each second. If the playback speed remains to be 30 fps, the end effect is a two-times slowdown of the motion in the video since the same motion captured in the 60 frames is being displayed across two seconds.”).
Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Flaks in view of Boorboor in further view of Xiang as rendering a video at 120 frames per second (fps) provides smooth motion, enhanced clarity for fast action, and crucial flexibility for creating high-quality slow-motion effects in post-production (See Xiang ¶3).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT J CRADDOCK whose telephone number is (571)270-7502. The examiner can normally be reached Monday - Friday 10:00 AM - 6 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT J CRADDOCK/Primary Examiner, Art Unit 2618