DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 5-6 are objected to because of the following informalities:
In claim 5, line 1, “The electronic device of claim 3” should read “The electronic device of claim 4” since in claim 5 the time parameter is introduced in claim 4 as “a time parameter”.
In claim 6, line 1, “The electronic device of claim 3” should read “The electronic device of claim 4” since in claim 5 the time parameter is firstly introduced in claim 4 as “a time parameter”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 7-15, 17-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Pointecker et al. (Bridging the Gap Across Realities: Visual Transitions Between Virtual and Augmented Reality, 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), DOI: 10.1109/ISMAR55827.2022.00101, hereinafter “Pointecker”) as in view of Peng et al. (US 20200007914 A1, hereinafter “Peng”).
Regarding claim 1, Pointecker discloses An electronic device comprising:
a display;
… memory; and
at least one processor,
… display a three-dimensional (3D) image in a first mode on the display; (page 828, col. left, para. 3, “Visual transitions are a common technique to change location or time in three-dimensional applications with stereoscopic HMDs (e.g. Portal [50])”; page 832, col. left, para. 2, “During the user study, the HMD was powered by a GeforceRTX 3090, an Intel Core i9-11900K, and 64 GB of RAM). Note that: (1) with stereoscopic HMDs a three-dimensional (3D) image can be displayed on the display in a first mode (VR mode) on the display; (2) the conventional main-stream HMD for AR / VR is an electronic device with a display; and (3) the HMD has RAM memory and processors in GeforceRTX 3090.
confirm a request to switch from the first mode to a second mode; (page 827, col. left, Abstract, “Modern video see-through head-mounted displays allow users to switch between augmented and virtual reality without removing the headset. This enables for the first time a fluent transition between augmented and virtual reality”; page 832, col. right, para. 1, “When users confirmed that everything was clear to them, they
were asked to initiate the first transition. The transition participants saw during this introduction was always the transition they would use in the first condition. Participants then received an introduction into all visualisations in the VR environment and then returned to AR”). Note that: the users were asked to initiate the first transition as a request. Then the user confirmed the request and switched from the first mode (the VR environment / mode) to a second mode (the AR mode).
determine whether the switching is possible within a specified time; (page 829 / col. right / para. 3 – page 830 / col. left / para. 1, “Initial testing has indicated that three seconds for the entire transition can be considered as comfortable”). Note that: three seconds can be regarded as a specific time for the user to determine that the switching is comfortable and possible for use.
generate a first intermediate image by processing a first image of the first mode and a second image of the second mode according to a first image processing method based on the switching being possible within the specified time; (page 829, col. right, para. 2, “Instead of immediately switching between two environments, the Fade transition gradually changes the surrounding environments. When moving from VR to AR, the transparency of all VR-related objects is increased over time, while the transparency of AR-related objects is decreased at the same level. The real world video stream also appears slowly as the VR environment is no longer able to hide the real world. The transition from AR to VR works vice versa.”; page 830, Figure 2: “Time-dependent representation from the use’s perspective during the transition from VR to AR. Fade (top) gradually increasing the transparency of VR-related objects and decreasing the transparency of AR-related objects”, and “
PNG
media_image1.png
202
846
media_image1.png
Greyscale
”; page 829, col. right, para. 3, “If the transition process is too fast, the effect is no longer noticeable. If the transition is too slow, the intermediate state in which both environments are visible to a similar extent will endure for a comparatively long time”). Note that: (1) “Fade” can be regarded as a first image processing method that processes a first image (the left image) of the first mode (the VR mode) and a second image (the right image) to generate a first intermediate image (the two middle images between the left image and the right image); (2) the intermediate images can be generated for a too fast transition process, and in this case the “Fade” is based on switching being possible within the specified time (3 seconds); and (3) the first image and the second image can be an image with a set of frames while the left image can be the last frame of the first image and the right image can be the start frame of the second image; and the first intermediate can be an image with a set of frames.
generate a second intermediate image by processing the first image of the first mode and the second image of the second mode according to a second image processing method based on the switching being possible beyond the specified time; and (page 829, col. right, para. 2, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other”; page 830, Figure 2: “(bottom) cuts away the VR environment using the clipping plane, revealing the AR environment.”, and “
PNG
media_image2.png
200
850
media_image2.png
Greyscale
”; page 829, col. right, para. 3, “If the transition process is too fast, the effect is no longer noticeable. If the transition is too slow, the intermediate state in which both environments are visible to a similar extent will endure for a comparatively long time”). Note that: (1) “SimpleCut” can be regarded as a second image processing method that processes the first image (the left image) of the first mode (the VR mode) and the second image (the right image) to generate a second intermediate image (the two middle images between the left image and the right image); and (2) the intermediate images can be generated for a too slow transition process, and in this case the “SimpleCut” is based on switching being possible beyond the specified time (3 seconds); the second intermediate image can be an image with a set of frames.
output the first intermediate image or the second intermediate image in a process of switching from the first mode to the second mode. (page 830, Figure 2: “Time-dependent representation from the use’s perspective during the transition from VR to AR. Fade (top) gradually increasing the transparency of VR-related objects and decreasing the transparency of AR-related objects”, and “
PNG
media_image1.png
202
846
media_image1.png
Greyscale
”). Note that: the intermediate image (two frames between the left image and the right image) can be output and displayed on the display of the HMD during the transition process from left image to the right image.
However, Pointecker fails to disclose, but in the same art of computer graphics, Peng discloses
… comprising processing circuitry, operatively connected to the display and the memory,
wherein the memory stores instructions and at least one processor, individually and/or collectively, is configured to execute the instructions and to cause the electronic device to: (Peng, para. [0006], “The electronic device includes at least one processor and a computer readable storage. The computer readable storage is coupled to the at least one processor and stores at least one computer executable instruction thereon which, when executed by the at least one processor, causes the at least one processor to:”). Note that: the processors, buses, and memory form a processing circuitry.
Pointecker and Peng, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply the electronic device includes at least one processor and a computer readable storage, as taught by Peng into Pointecker. The motivation would have been “The electronic device includes at least one processor and a computer readable storage. The computer readable storage is coupled to the at least one processor and stores at least one computer executable instruction thereon which, when executed by the at least one processor, causes the at least one processor to:” (Peng, para. [0006]). The suggestion for doing so would allow to use an electronic data processing device to perform computer graphics operations. Therefore, it would have been obvious to combine Pointecker and Peng.
Regarding claim 2, Pointecker in view of Peng discloses The electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to generate the first intermediate image or the second intermediate image using a first object displayed in the first mode or a second object displayed in the second mode. (Pointecker, page 830, Figure 2: “Time-dependent representation from the use’s perspective during the transition from VR to AR. Fade (top) gradually increasing the transparency of VR-related objects and decreasing the transparency of AR-related objects”, and “
PNG
media_image1.png
202
846
media_image1.png
Greyscale
”). Note that: (1) the electronic device generates the first intermediate image (one of the two frames between the left image in the first mode (VR) and the right image in the second mode (VR); and (2) one of four standing case-liked objects in the left image can be regarded as a first object used and displayed in the first mode (VR), while the storage furniture in the right image can be regarded as a second object used and displayed in the second mode (AR).
Regarding claim 7, Pointecker in view of Peng discloses The electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to generate the first intermediate image by an alpha blending method. (Pointecker, page 892, para. 2, “When moving from VR to AR, the transparency of all VR-related objects is increased over time, while the transparency of AR-related objects is decreased at the same level. The real world video stream also appears slowly as the VR environment is no longer able to hide the real world. The transition from AR to VR works vice versa.”). Note that: the “Fade” process is a conventional alpha blending method by changing the transparence of VR- and AR-related objects to form an image or frame.
Regarding claim 8, Pointecker in view of Peng discloses The electronic device of claim 7, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to:
generate one frame of the first intermediate image; Note that: (1) the first intermediate image can include a set of intermediate frames; (2) it is obvious to one having ordinary skill in the art that one frame (Ifirst_immediate,1) of the first intermediate image can be generated by alpha blending in claim 7 using the start frame of the second image and the last frame of the first image (e.g., Ifirst_intermediate,1 = Alpha1*Ifirst_image,last + (1-Alpha1)* Isecond_image,start) where Ifirst_image,last is the end or last frame of the first image and Isecond_image,start is the start frame of the second image.
perform first alpha blending of a last frame of the first image and the one frame; Note that: a second frame (Ifirst_immediate,2) of the first intermediate image can be generated by alpha blending in claim 7 using the one frame and the last frame of the first image (e.g., Ifirst_intermediate,2 = Alpha2*Ifirst_image,last + (1-Alpha2)* Ifirst_intermediate,1) where Ifirst_image,last is the last frame of the first image and Ifirst_intermediate,1 is the one frame.
perform second alpha blending of the one frame and a start frame of the second image; and Note that: a third frame (Ifirst_intermediate,3) of the first intermediate image can be generated by alpha blending in claim 7 using the one frame and the start frame of the second image (e.g., Ifirst_intermediate,3 = Alpha2*Isecond_image,start + (1-Alpha2)* Ifirst_intermediate,1) where Ifirst_image,last is the last frame of the first image and Ifirst_intermediate,1 is the one frame.
generate the first intermediate image by the first and second alpha blending. Note that: the first intermediate image including frames Ifirst_intermediate,3 Ifirst_intermediate,1 Ifirst_intermediate,2 is generated by the first and second alpha blending.
Regarding claim 9, Pointecker in view of Peng discloses The electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to generate and output the first intermediate image or the second intermediate image having a plurality of frames that change with time. (Pointecker, page 892, para. 2, “When moving from VR to AR, the transparency of all VR-related objects is increased over time, while the transparency of AR-related objects is decreased at the same level. The real world video stream also appears slowly as the VR environment is no longer able to hide the real world. The transition from AR to VR works vice versa.”). Note that: (1) the “Fade” process is a conventional alpha blending method by changing the transparence of VR- and AR-related objects. (2) the first intermediate image can include a set of intermediate frames; (3) it is obvious to one having ordinary skill in the art that one frame (Ifirst_immediate,1) of the first intermediate image can be generated by alpha blending in claim 7 using the start frame of the second image and the last frame of the first image (e.g., Ifirst_intermediate,1 = Alpha1*Ifirst_image,last + (1-Alpha1)* Isecond_image,start) where Ifirst_image,last is the end or last frame of the first image and Isecond_image,start is the start frame of the second image; (4) a second frame (Ifirst_intermediate,2) of the first intermediate image can be generated by alpha blending in claim 7 using the one frame and the last frame of the first image (e.g., Ifirst_intermediate,2 = Alpha2*Ifirst_image,last + (1-Alpha2)* Ifirst_intermediate,1) where Ifirst_image,last is the last frame of the first image and Ifirst_intermediate,1 is the one frame; (5) a third frame (Ifirst_ intermediate,3) of the first intermediate image can be generated by alpha blending in claim 7 using the one frame and the start frame of the second image (e.g., Ifirst_ intermediate,3 = Alpha2*Isecond_image,start + (1-Alpha2)* Ifirst_intermediate,1) where Ifirst_image,last is the last frame of the first image and Ifirst_intermediate,1 is the one frame; (6) the first intermediate image including frames Ifirst_ intermediate,3 Ifirst_ intermediate,1 Ifirst_ intermediate,2 is generated by the first and second alpha blending; and (7) the second intermediate image including a plurality of frames can be generated in the similar method to that for the first intermediate image including a plurality of frames, while Ifirst_ intermediate,3 Ifirst_ intermediate,1 Ifirst_ intermediate,2 are generated with the corresponding time.
Regarding claim 10, Pointecker in view of Peng discloses The electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to store information about an entity including a start frame of the second image in the memory. (Pointecker, page 832, col. left, para. 2, “During the user study, the HMD was powered by a GeforceRTX 3090, an Intel Core i9-11900K, and 64 GB of RAM, resulting in an average of 80 frames per second. For input, we used the HTC Vive handheld controller, where all transitions were triggered by participants using the grip button on the side of the controller. To select virtual buttons, such as the filter buttons, or user study related buttons, we used position based input from the virtual controller representation. Participants were able to move around a 4x4m tracking room with a table as passive haptic feedback”; page 829, col. right, para. 2, “Instead of immediately switching between two environments, the Fade transition gradually changes the surrounding environments. When moving from VR to AR, the transparency of all VR-related objects is increased over time, while the transparency of AR-related objects is decreased at the same level. The real world video stream also appears slowly as the VR environment is no longer able to hide the real world. The transition from AR to VR works vice versa.”). Note that: the memory of the HMD powered by a GeforceRTX 3090, an Intel Core i9-11900K, and 64 GB of RAM can store information about the HDM device or its VR/AR module as an entity, and VR/AR modes’ content information for blending the first image and the second image including the start frame of the second image.
Regarding claim 11, Pointecker in view of Peng discloses The electronic device of claim 1, wherein the first mode is an augmented reality (AR) mode and the second mode is a virtual reality (VR) mode. (Pointecker, page 829, col. right, para. 2, “Instead of immediately switching between two environments, the Fade transition gradually changes the surrounding environments. When moving from VR to AR, the transparency of all VR-related objects is increased over time, while the transparency of AR-related objects is decreased at the same level. The real world video stream also appears slowly as the VR environment is no longer able to hide the real world. The transition from AR to VR works vice versa.”). Note that: (1) the blending method of VR/AR for the transition can work transition from AR to VR works vice versa; (2) in the later case, the first mode is an augmented reality (AR) mode and the second mode is a virtual reality (VR) mode, and this can happen when the user switch from AR to VR in his whole experience.
Regarding claim 12, Pointecker in view of Peng discloses The electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to:
recognize a real object included in the first image; and (Pointecker, page 830, col. left, paras. 3-4, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other … hatches are drawn on top of the sectional area, similar to an engineering drawing that provides information about the inside of an object. To clarify the transition process and draw the user’s attention, small sparks are emitted at the cutting plane. These sparks help to easily identify where the dividing line between the VR and AR environment is located”; page 829, col. right, para. 2, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other”; page 830, Figure 2: “(bottom) cuts away the VR environment using the clipping plane, revealing the AR environment.”, and “
PNG
media_image2.png
200
850
media_image2.png
Greyscale
”). Note that: a real object (wall) in the first image (the right image) is recognized by drawing hatches for defining the split plane for the second image processing (“SimpleCut transition”).
display an object that has the same shape as the real object and in which style of the second mode is reflected, in the second intermediate image. (Pointecker page 829, col. right, para. 2, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other”; page 830, Figure 2: “(bottom) cuts away the VR environment using the clipping plane, revealing the AR environment.”, and “
PNG
media_image2.png
200
850
media_image2.png
Greyscale
”). Note that: (1) two images or frames between the first image on the right (AR, start frame) and the second image on the right (VR, end frame) are generated and displayed on the HMD’s display; and (2) the frame (the middle-left) in the second intermediate image (the two frames between the left and the right) shows the real object (wall) in the same shape but with the style of mirror wall of the second mode (VR) on the left.
Regarding claim 13, Pointecker in view of Peng discloses The electronic device of claim 12, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to:
detect a display area of the real object; and (Pointecker, page 830, col. left, paras. 3-4, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other … hatches are drawn on top of the sectional area, similar to an engineering drawing that provides information about the inside of an object. To clarify the transition process and draw the user’s attention, small sparks are emitted at the cutting plane. These sparks help to easily identify where the dividing line between the VR and AR environment is located”; page 829, col. right, para. 2, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other”; page 830, Figure 2: “(bottom) cuts away the VR environment using the clipping plane, revealing the AR environment.”, and “
PNG
media_image2.png
200
850
media_image2.png
Greyscale
”). Note that: (1) a real object (wall) in the first image (the right image) is recognized by drawing hatches for defining the split plane for the second image processing (“SimpleCut transition”); and (2) the recognized real wall can be regarded as a detected area.
reflect the style of the second mode in the display area. (Pointecker page 829, col. right, para. 2, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other”; page 830, Figure 2: “(bottom) cuts away the VR environment using the clipping plane, revealing the AR environment.”, and “
PNG
media_image2.png
200
850
media_image2.png
Greyscale
”). Note that: (1) two images or frames between the first image on the right (AR, start frame) and the second image on the right (VR, end frame) are generated and displayed on the HMD’s display; and (2) the frame (the middle-left) in the second intermediate image (the two frames between the left and the right) shows the style of mirror wall of the second mode (VR) on the left.
Regarding claim 14, Pointecker in view of Peng discloses The electronic device of claim 1, wherein the first mode is a first VR mode, and the second mode is a second VR mode. (Pointecker, page 829, col. right, “Fade”, “VR-VR”),
PNG
media_image3.png
464
474
media_image3.png
Greyscale
”). Note that: for “Fade” image processing, the transition can be from VR to VR (VR-VR). In this case, the first mode is VR while the second mode is VR.
Regarding claim 15, Pointecker in view of Peng discloses The electronic device of claim 1, wherein the first mode is a VR mode, and the second mode is an AR mode. (Pointecke, page 830, Figure 2: “Time-dependent representation from the use’s perspective during the transition from VR to AR. Fade (top) gradually increasing the transparency of VR-related objects and decreasing the transparency of AR-related objects”, and “
PNG
media_image1.png
202
846
media_image1.png
Greyscale
”).Note that: for the “Fade” image processing here, the transition is from VR to AR.
Regarding claim 17, Pointecker discloses An electronic device comprising:
a display;
a memory; and at least one processor, comprising processing circuitry, operatively connected to the display and the memory,
… display a three-dimensional (3D) image in a first mode on the display; (page 828, col. left, para. 3, “Visual transitions are a common technique to change location or time in three-dimensional applications with stereoscopic HMDs (e.g. Portal [50])”; page 832, col. left, para. 2, “During the user study, the HMD was powered by a GeforceRTX 3090, an Intel Core i9-11900K, and 64 GB of RAM). Note that: (1) with stereoscopic HMDs a three-dimensional (3D) image can be displayed on the display in a first mode (AR mode) on the display; (2) the conventional main-stream HMD for AR / VR is an electronic device with a display; and (3) the HMD has RAM memory and processors in GeforceRTX 3090.
confirm a request to switch from the first mode to a second mode; (page 827, col. left, Abstract, “Modern video see-through head-mounted displays allow users to switch between augmented and virtual reality without removing the headset. This enables for the first time a fluent transition between augmented and virtual reality”; page 832, col. right, para. 1, “When users confirmed that everything was clear to them, they were asked to initiate the first transition. The transition participants saw during this introduction was always the transition they would use in the first condition.”). Note that: the users were asked to initiate the first transition as a request. Then the user confirmed the request and switched from the first mode (the AR environment / mode) to a second mode (the VR mode).
generate at least one virtual object related to the second mode; and (page 831, col. left, para.1, “The portal metaphor imitates a gateway to another reality. In our first approach the portal could be positioned freely in space, depending on the position and orientation of controllers. Once the portal is open, the transition requires physically walking through the portal to switch between realities”; Figure 3: “Time-dependent representation from the users perspective during … Portal (bottom) from AR to VR”, the bottom-left image (AR’s last frame) has a virtual black ball with its shadow (e.g., a virtual model of the earth) generated as one virtual object on the floor in AR, the bottom-right image (VR’s start frame) has the virtual shadow on the floor, and “
PNG
media_image4.png
236
1016
media_image4.png
Greyscale
”). Note that: the one virtual object (virtual black ball’s shadow) is related to the start frame of the second mode (the VR) with a shadow on the floor, and its shadow shows the two transition frames between the left image (AR) and the right image (VR).
add and output the at least one virtual object to a first image in the first mode in a process of the switching from the first mode to the second mode. Note that: the one virtual object (the black ball with its shadow) is added and output to a first image that includes the two transition / intermediate frames.
However, Pointecker fails to disclose, but in the same art of computer graphics, Peng discloses
comprising processing circuitry, operatively connected to the display and the memory, wherein the memory stores instructions and at least one processor, individually and/or collectively, is configured to execute the instructions and to cause the electronic device to: (Peng, para. [0006], “The electronic device includes at least one processor and a computer readable storage. The computer readable storage is coupled to the at least one processor and stores at least one computer executable instruction thereon which, when executed by the at least one processor, causes the at least one processor to:”). Note that: the processor, buses, and memory form a processing circuitry
The motivation to combine Pointecker and Peng given in claim 1 is incorporated here.
Regarding claim 18, Pointecker and Peng discloses The electronic device of claim 17, wherein at least one processor, individually and/or collectively, configured to cause the electronic device to add the at least one virtual object to the first image by reflecting an arrangement form of a real object displayed in the first mode. (Pointecker, page 831, col. left, para.1, “The portal metaphor imitates a gateway to another reality. In our first approach the portal could be positioned freely in space, depending on the position and orientation of controllers. Once the portal is open, the transition requires physically walking through the portal to switch between realities”; Figure 3: “Time-dependent representation from the users perspective during … Portal (bottom) from AR to VR”, the bottom-left image (AR’s last frame) has a virtual black ball with its shadow generated as one virtual object on the floor in AR, the bottom-right image (VR’s start frame) has the virtual shadow on the floor, and “
PNG
media_image4.png
236
1016
media_image4.png
Greyscale
”). Note that: (1) the one virtual object (the black ball with its shadow) is added and output to the first image that includes the two transition / intermediate frames; and (2) the virtual black ball and its shadow (e.g., a virtual model of the earth) is added in the middle of a room with the real objects (desk and chairs) reflecting an arrangement of a study room.
Claim 20 reciting “A method of displaying a three-dimensional image that is performed on an electronic device, the method comprising:” is corresponding to claim 1. Therefore, claim 20 is rejected for the same rationale for claim 1.
In addition, Pointecker in view of Peng disclose A method of displaying a three-dimensional image that is performed on an electronic device, the method comprising: (page 828, col. left, para. 3, “Visual transitions are a common technique to change location or time in three-dimensional applications with stereoscopic HMDs (e.g. Portal [50])”; page 832, col. left, para. 2, “During the user study, the HMD was powered by a GeforceRTX 3090, an Intel Core i9-11900K, and 64 GB of RAM). Note that: (1) with stereoscopic HMDs a three-dimensional (3D) image can be displayed on the display in a first mode (VR mode) on the display; (2) the conventional main-stream HMD for AR / VR is an electronic device with a display; and (3) the HMD has RAM memory and processors in GeforceRTX 3090; and (4) Visual transitions as a common technique is a method.
Claims 3-4 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Pointecker in view of Peng and Xiao et al. (Multi-Scale Attention Generative Adversarial Networks for Video Frame Interpolation, Digital Object Identifier, IEEE Access, 10.1109/ACCESS.2020.2995705, hereinafter “Xiao”).
Regarding claim 3, Pointecker in view of Peng discloses The electronic device of claim 1, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to generate the second intermediate image using (Pointecker, page 829, col. right, para. 2, “During the SimpleCut transition, the scene is cut with an invisible clipping plane that runs through the entire environment from one side to the other”; page 830, Figure 2: “(bottom) cuts away the VR environment using the clipping plane, revealing the AR environment.”, and “
PNG
media_image2.png
200
850
media_image2.png
Greyscale
”). Note that: the second intermediate image is generated using “SimpleCut”.
However, Pointecker in view of Peng fails to disclose, but in the same art of computer graphics, discloses a generative Al. (Xiao, page 94842, Abstract, “multi-scale dense attention generative adversarial network is proposed. First, a multi-scale generative adversarial framework is established for video frame interpolation. Generators from coarse to fine can better combine global and local information. Second, an attention module introduced to generator makes network accurately focus on moving objects. Third, a sequence discriminator is designed to improve the ability of capturing spatial and temporal consistency in frame sequence.”). Note that: (1) Multi-Scale Attention Generative Adversarial Networks as a generative AI can generate a frame image (the second intermediate image) from two frames; (2) the generative AI can substitute “SimpleCut” to generate the second intermediate image above.
Pointecker in view of Peng, and Xiao, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply Multi-Scale Attention Generative Adversarial Network as a generative AI to generate intermediate frames, as taught by Xiao into Pointecker in view of Peng. The motivation would have been “our approach attains higher performance and produce more photo-realistic in-between frame comparing with previous works” (Xiao, page 94842, Abstract). The suggestion for doing so would allow to generate an intermediate image or frame using a generative AI. Therefore, it would have been obvious to combine Pointecker, Peng, and Xiao.
Regarding claim 4, the combination of Pointecker, Peng, and Xiao discloses The electronic device of claim 3, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to input a time parameter to the generative Al to generate the second intermediate image. (Xiao, page 94842, col. left, para. 2, “Video frame interpolation can be used in numerous applications, such as video frame rate conversion [1], slow-motion video generation [2]”; page 94846, col. left, para. 2, “The consecutive two real frames and the generated intermediate frames compose a fake sequence, Fseq D contact(I1; IS ; I2), and DS distinguishes real sequence from fake sequence”). Note that: (1) the interpolation between two consecutive frames can be repeated and applied for slow motion processing by generating a set of interpolated frames: a) generate an interpolated image (It+0.5) at timestamp t+0.5*Delta between two consecutive two real frames It and It+1*Delta at timestamps t and t+1.0*Delta where Delta is an time parameter for the transition; b) inputting a timestamp (e.g., t+0.25*Delta) related to the time parameter Delta to the method above, the generative AI can generate an interpolated frame It+0.25*Delta between the frame It and the interpolated frame It+0.5*Delta ; c) repeat step b), by inputting the timestamps (e.g., t+0.125*Delta, t+0.375*Delta, t+0.5*Delta, t+0.625*Delta, t+0.875*Delta …), a set of interpolated frames It+0.125*Delta, It+0.375*Delta, It+0.5*Delta, It+0.625*Delta, It+0.875*Delta , … can be generated according to the corresponding input time; and (2) at a certain input time, the corresponding interpolated frame can be regarded as the second intermediate image.
The motivation to combine Pointecker, Peng, and Xiao given in claim 3 is incorporated here.
Regarding claim 6, the combination of Pointecker, Peng, and Xiao discloses The electronic device of claim 3, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to generate the second intermediate image including a number of frames proportional to the time parameter. (Xiao, page 94842, col. left, para. 2, “Video frame interpolation can be used in numerous applications, such as video frame rate conversion [1], slow-motion video generation [2]”; page 94846, col. left, para. 2, “The consecutive two real frames and the generated intermediate frames compose a fake sequence, Fseq D contact(I1; IS ; I2), and DS distinguishes real sequence from fake sequence”). Note that: (1) as given in claim 4, the method’s steps a)-c) can be applied to the first image and the second image with the inputting time parameter to generate a set of interpolated intermediate frames that can be regarded as whole as the second intermediate image that include the set of interpolated frames as a plurality of frames (frames It+0.125*Delta, It+0.375*Delta, It+0.5*Delta, It+0.625*Delta, It+0.875*Delta , …); and (2) it is obvious to one having ordinary skill in the art that the timestamps can be split proportionally to the time parameter Delta for the interpolated frames’ timestamps (e.g., t+0.125*Delta, t+0.375*Delta, t+0.5*Delta, t+0.625*Delta, t+0.875*Delta …).
The motivation to combine Pointecker, Peng, and Xiao given in claim 3 is incorporated here.
Claims 5 are rejected under 35 U.S.C. 103 as being unpatentable over Pointecker in view of Peng, Xiao, and Gil et al. (US 10,055,887 B1, hereinafter “Gil”).
Regarding claim 5, Pointecker in view of Peng and Xiao discloses The electronic device of claim 3, wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to:
However, Pointecker in view of Peng and Xiao fails to disclose, but in the same art of computer graphics, Gil discloses determine the time parameter based on at least one of an illuminance difference between the first image and the second image, a structural difference between the first mode and the second mode, and a loading time of the second mode. (Gil, col. 9, lines 9-43, “These preset intervals of time may be, for example, a minimum interval of time determined to be sufficient to make an effective transition from the virtual brightness and/or sound level to the ambient brightness and/or sound level for a particular magnitude of difference between the respective virtual and ambient levels. Thus, the preset interval may vary, depending on the magnitude of the difference(s). For example, if a relatively large difference in brightness is detected between the virtual environment and the ambient environment, then the preset interval of time provided for adjusting the brightness level of the virtual environment toward the brightness level of the ambient may be longer than what would be experienced for a smaller difference in brightness … the HMD 100 may be equipped with a camera 180 as described above, and thus may be capable of operating in the pass through mode as described above. In this embodiment, the transition from the virtual environment to the ambient environment may be made without moving the front portion of the HMD 100 out of the line of sight, or field of view, of the user”). Note that: (1) between two displayed images (the first image and the second image) of two modes (from VR to AR) on the HMD, the brightness or illuminance difference of the images may change abruptly because the ambient light by using the see-through optical screen. The preset interval may vary, depending on the magnitude of the difference(s); (2) the preset interval (transition time parameter) is determined or adjusted: if a relatively large difference in brightness is detected between the virtual environment’s displayed image (e.g., VR) and the ambient environment’s displayed image (e.g., AR), then the preset interval of time provided for adjusting the brightness level of the virtual environment toward the brightness level of the ambient may be longer than what would be experienced for a smaller difference in brightness; (3) as clearly specified in the specification, “As used herein, each of such phrases as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C," may include any one of, or all possible combinations of the items enumerated together in a corresponding one of the phrases” (the Specification, page 35, lines 2-5). Therefore, Gil discloses the limitations of claim 5.
Pointecker in view of Peng and Xiao, and Gil, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply adjusting or determining transition interval time (time parameter) based on an illuminance or brightness difference between the images, as taught by Gil into Pointecker in view of Peng and Xiao. The motivation would have been “the preset interval may vary, depending on the magnitude of the difference(s). For example, if a relatively large difference in brightness is detected between the virtual environment and the ambient environment, then the preset interval of time provided for adjusting the brightness level of the virtual environment toward the brightness level of the ambient may be longer than what would be experienced for a smaller difference in brightness” (Gil, col. 9, lines 14-22). The suggestion for doing so would allow to adjust or determine transition interval time (time parameter) based on an illuminance or brightness difference between the images. Therefore, it would have been obvious to combine Pointecker, Peng, Xiao, and Gil.
Claims 16 are rejected under 35 U.S.C. 103 as being unpatentable over Pointecker in view of Peng and Black et al. (US 2020/0238177 A1, hereinafter “Black”).
Regarding claim 16, Pointecker in view of Peng discloses The electronic device of claim 1, further comprising a sensor and a camera,
wherein at least one processor, individually and/or collectively, is configured to cause the electronic device to: (Pointecker, page 827, col. right, 2, “consumer-grade HMDs are equipped with high-resolution front-facing cameras with a large FoV and acceptable latency. Users are able to move freely along the RV continuum without having to remove the HMD.”). Note that: the a conventional DMD can have camera and other sensors (e.g., accelerators, microphones, LiDAR).
generate and output the first intermediate image (Pointecker, page 829, col. right, para. 2, “Instead of immediately switching between two environments, the Fade transition gradually changes the surrounding environments. When moving from VR to AR, the transparency of all VR-related objects is increased over time, while the transparency of AR-related objects is decreased at the same level. The real world video stream also appears slowly as the VR environment is no longer able to hide the real world. The transition from AR to VR works vice versa.”; page 830, Figure 2: “Time-dependent representation from the use’s perspective during the transition from VR to AR. Fade (top) gradually increasing the transparency of VR-related objects and decreasing the transparency of AR-related objects”, and “
PNG
media_image1.png
202
846
media_image1.png
Greyscale
”; page 829, col. right, para. 3, “If the transition process is too fast, the effect is no longer noticeable. If the transition is too slow, the intermediate state in which both environments are visible to a similar extent will endure for a comparatively long time”; page 832, col. right, papa. 1, “When users confirmed that everything was clear to them, they were asked to initiate the first transition.”). Note that: (1) “Fade” can be regarded as a first image processing method that processes a first image (the left image) of the first mode (the VR mode) and a second image (the right image) to generate a first intermediate image (the two middle images between the left image and the right image); (2) the intermediate images can be generated for a too fast transition process, and in this case the “Fade” is based on switching being possible within the specified time (3 seconds); and (3) the users are asked to initiate or trigger the transition process.
However, Pointecker in view of Peng fails to disclose, but in the same art of computer graphics, Black discloses
divide an area around the electronic device into a plurality of zones; (Black, page 1, para. [0009], “The method also includes identifying the real-world space and a safe zone within the real-world space for interacting with a virtual reality space via the HMD”; page 1, para. [0016], “FIGS. 4A-4B illustrates an example of the user approaching an outer boundary of the real world space and the providing of guidance back to the safe zone, in accordance with one embodiment”; FIG. 1: the extended device including device “102” (e.g., a HMD), “COMPUTER” 106, and “CAMERA” 108 can be regarded as the electronic device, “
PNG
media_image5.png
520
680
media_image5.png
Greyscale
”). Note that: (1) the real area around the user is divided into at least two zone (i.e., safe zone and the zone outside the safe zone) as a plurality of zones; (2) the extended device determines the zones monitored by camera 108.
detect whether the electronic device enters a designated zone using the sensor and/or the camera; and (Black, page 2, para. [0030], “camera 108 can be configured to capture images of the interactive environment in which the user 100 is located. These captured images can be analyzed to determine the location and movements of the user 100, the HMD 102, and the glove interface object 104a”). Note that: (1) the location of the part of electronic device (HMD 102) worn by the user is analyzed and determined; and (2) the part of electronic device can be determined whether (HMD 102) enters a designated zone (i.e., an unsafe zone) by checking the location of HMD 102 is within the zones or not.
generate and output the first intermediate image based on the electronic device entering the designated zone. (FIG. 5A: a amination of “Backup!” is displayed’ “
PNG
media_image6.png
396
550
media_image6.png
Greyscale
; page 4, para. [0051], “FIG. SA illustrates an example where the user 100 is provided with more animated characters 504 in the virtual-reality space, to provide guidance back into the safe zone, in accordance with one embodiment. As shown, the user 100 has wandered off to the right of the safe zone, and is approaching the coffee table 304 and the lamp 306”). Note that: (1) the animation of “Backup!” displayed , the user cam move back into the safe zone; and (2) the user’s movement to the safe zone can substitute “asking the user” to trigger the user to start the transition, and the first immediate image are generated accordingly above.
Pointecker in view of Peng, and Black, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply making a plurality of zones for safety, detecting the location and the even whether a part of device or a user enters a designated a zone, and generating a animation message to guide the users to act accordingly as a trigger, as taught by Black into Pointecker in view of Peng. The motivation would have been “The method includes integrating content into the virtual reality space. The content is configured to provide guidance in direction of movement of the user toward the safe zone of the real-world space. If the user continues to move away from safe zone, pausing presentation of the virtual reality space, and resuming the presentation when the user is in the safe zone.” (Black, Abstract). The suggestion for doing so would allow to set zones, monitor users related to the zones, and guide the users back into a safe zone when user enters into a unsafe zone as a trigger to generate the intermediate image for mode transition. Therefore, it would have been obvious to combine Pointecker, Peng, and Black.
Claims 19 is rejected under 35 U.S.C. 103 as being unpatentable over Pointecker in view of Peng and Wiki_Progress_indicator (Progress indicator, archive.org, https://web.archive.org/web/20231123032829/https://en.wikipedia.org/wiki/Progress_indicator, hereinafter “Wiki_Progress_indicator”).
Regarding claim 19, Pointecker and Peng discloses The electronic device of claim 17, wherein the at least one virtual object is plural in number, and
… at least one processor, individually and/or collectively, is configured to cause the electronic device (Pointecker, Figure 3: “Time-dependent representation from the users perspective during … Portal (bottom) from AR to VR”, the bottom-left image (AR’s last frame) has a virtual black ball with its shadow generated as one virtual object on the floor in AR, the bottom-right image (VR’s start frame) has the virtual shadow on the floor, and “
PNG
media_image4.png
236
1016
media_image4.png
Greyscale
”). Note that: the ceiling light, the black ball’s shadow on the floor, and the panel wall in the right image (VR) can be regarded as the at least one virtual object plural in number.
… to increase the number of the at least one virtual object added to the first image according to a transition timestamp corresponding to a transition interval. Note that: the black ball’s shadow and the panel wall are added to the first image including two frames between the left image and the right image gradually by increasing the number the at least one virtual object from 1 to 2 according to a transition timestamp corresponding to a transition interval shown in Figure 3 above.
However, Pointecker in view of Peng fails to disclose, but in the same art of computer graphics, Wiki_Progress_indicator discloses
a loading progress state of an application of the second mode. (Wiki_Progress_indicator, page 1, para. 1, “A progress indicator is an element of a command-line interface, a textual user interface, or a graphical user interface that is intended to inform the user that an operation is in progress, to reassure that the system is not hung or waiting for user input, and often to provide the user with an estimate of how far through a task the system has progressed. Page 1, para. 2, “A progress bar, a typically horizontal bar which is gradually filled with a color as the process completes.”). Note that: (1) “A progress bar” can substitute a transition timestamp corresponding to a transition interval to synchronize the increase of the number of the number of the at least one virtual object; and (2) an application of the second mode is an AR application as a program, and the loading progress can be regarded as an operation progress.
Pointecker in view of Peng, and Wiki_Progress_indicator, are in the same field of endeavor, namely computer graphics. Before the effective filing date of the claimed invention, it would have been obvious to apply using a progress bar to indicate the application loading status, as taught by Wiki_Progress_inidcator into Pointecker in view of Peng. The motivation would have been “is intended to inform the user that an operation is in progress, to reassure that the system is not hung or waiting for user input, and often to provide the user with an estimate of how far through a task the system has progressed” (Wiki_Progress_inidcator, page 1, para. 1). The suggestion for doing so would allow to inform the user that an operation is in progress, to reassure that the system is not hung or waiting for user input. Therefore, it would have been obvious to combine Pointecker, Peng, and Wiki_Progress_inidcator.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Pointecker et al. (Exploration of Visual Transitions Between Virtual and Augmented Reality, Immersive Analytics Workshop at Honolulu ’20, April 25–26, 2020, Honolulu, HI) teaches “Especially in immersive analytics, techniques for seamless transitions between VR and AR would greatly increase the visualisation and collaboration possibilities.”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BIAO CHEN whose telephone number is (703)756-1199. The examiner can normally be reached M-F 8am-5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Biao Chen/
Patent Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights