Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 7, 15, 16, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Dai et al. (US 2020/0204783, hereinafter Dai) in view of Chu et al. (US 2020/0092599, hereinafter Chu).
Regarding claim 1, Dai et al., teaches an image processing method, applied to a server in communication connection with a Virtual Reality (VR) terminal device ([0035] “a method for processing image data applied to a sending end of a host… of a virtual reality device, …for a host that conducts data transmission with a head-mounted display wirelessly”),
the method comprising: acquiring a first rendered frame and a second rendered frame ([0037], and Fig. 1, S110, “acquiring a left eye image data frame and a corresponding right eye image data frame that are to be transmitted”)
acquiring difference data from the second rendered frame, wherein the difference data is from an area in the second rendered frame which is not repeated in the first rendered frame; ([0037], and Fig. 1, S110, “calculating a difference between the acquired left eye image data frame and corresponding right eye image data frame, to obtain difference information between the two eye image data frames”)
and sending the first rendered frame and the difference data to the VR terminal device ([0041] and Fig. 1, S130, “sending the difference encoded information and the left eye image data frame to a head-mounted display of the virtual reality device.”)
such that the VR terminal device restores the second rendered frame according to the first rendered frame and the difference data ([0054] “The head-mounted display… can construct the right eye image data frame by conducting simple addition operations with respect to the difference information and the received left eye image data frame”)
and obtains a display image according to the first rendered frame and the second rendered frame ([0054] “and can output and display the image data after the right eye image data frame is completely constructed.”).
Dai fails to teach the method applied to a cloud server, wherein the cloud server is equipped with a first Field of View (FOV) rendering camera and a second FOV rendering camera, and wherein the first rendered frame is from the first FOV rendering camera, and the second rendered frame is from the second FOV rendering camera.
However, it is known in the art as taught by Chu. Chu teaches the method applied to a cloud server ([0055] “A client-server communication flow for a standard cloud gaming platform”).
wherein the cloud server is equipped with a first Field of View (FOV) rendering camera and a second FOV rendering camera ([0117] “the server device renders … between a left eye and right eye of the user”. Note: a rendering camera is a virtual device in a VR system, where it renders an image captured in a virtual scene),
and wherein the first rendered frame is from the first FOV rendering camera, and the second rendered frame is from the second FOV rendering camera ([0101] “the stereoscopic display utilizes two rendered viewports for display, one for the left eye of the user and one for the right eye of the user.” Note: the FOV rendering cameras correspond to the user’s eyes).
Chu is analogous to the claimed invention, as both relate to stereoscopic rendering for head-mounted displays. Chu further teaches that by stereoscopic rendering through the server, “client computing devices can utilize the high-end graphics provided by powerful server GPUs, allowing the users to enjoy high-end graphics on less powerful client computing devices” [0002]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chu to Dai, to offset the computing load of rendering the first and second rendered frame to the cloud server, rather than the VR device itself.
Regarding claim 7, claim 7 recites substantially similar limitations to claim 1, but in view of the VR terminal device instead of the cloud server. In addition, the combination of Dai and Chu teaches an image processing method, applied to a Virtual Reality (VR) terminal device in communication connection with a cloud server (Dai; [0046] “a method for processing image data applied to a head-mounted display… of a virtual reality device… that conducts data transmission with a host”).
Regarding claim 15, using the rationale of claim 1, the combination of Dai and Chu teaches a cloud server and the image processing method of claim 1. The combination of Dai and Chu further teaches a cloud server, comprising: a memory, a processor, (Dai; [0073] “As shown in FIG. 5, the device for processing image data comprises a machine-readable storage medium 51 and a processor 52”, where “FIG. 5 is a structural block diagram of a device for processing image data applied to a sending end of a host according to an embodiment of the present disclosure” [0073])
and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, causes the processor to perform the image processing method of claim 1 (Dai; [0073] “the machine-readable storage medium 51 stores a computer program executable by the processor 52, and when executed by the processor 52 the computer program implements the following steps”, where the following steps are the steps as shown in FIG. 1).
Regarding claim 16, using the rationale of claim 7, the combination of Dai and Chu teaches a Virtual Reality (VR) terminal device and the image processing method of claim 7. The combination of Dai and Chu further teaches a Virtual Reality (VR) terminal device, comprising: a memory, a processor, (Dai; [0079] “As shown in FIG. 6, the device for processing image data comprises a machine-readable storage medium 61 and a processor 62”, where “FIG. 6 is a structural block diagram of a device for processing image data applied to a head-mounted display according to an embodiment of the present disclosure.” [0079])
and a computer program stored in the memory and executable by the processor, (Dai; [0079] “the machine-readable storage medium 61 stores a computer program executable by the processor 62”)
wherein the computer program, when executed by the processor, causes the processor to perform the image processing method of claim 7 (Dai; [0079] “and when executed by the processor 62 the computer program implements the following steps”).
In regards to claim 17, claim 17 recites substantially similar limitations to claim 1, but in a medium form. The combination of Dai and Chu further teaches a non-transitory computer-readable storage medium, storing a computer-executable instruction which, when executed by a processor, causes the processor to perform an image processing method (Dai; [0023] “a host of a virtual reality device comprises a machine-readable storage medium… [where] the machine-readable storage medium stores a computer program executable by the processor, and when executed by the processor the computer program implements the steps of the above method.”).
Claims 2, 5, 8, 13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Dai et al. (US 2020/0204783) in view of Chu et al. (US 2020/0092599), and further in view of Desaulniers (“Stereoscopic Rendering in WebVR”) and Chen et al. (US 2020/0265618).
In regards to claim 2, the combination of Dai and Chu teaches the method of claim 1, wherein before acquiring a first rendered frame and a second rendered frame, the method further comprises: acquiring information sent by the VR terminal device (Chu; [0036] “the client device 12 gathers client-side user input and forwards that user input over the network to the server device 14, which in turn renders content based on the user input”). Chu teaches that this is done as the “input at the client computing device is sent to the server and influences the rendered content” [0003]. Therefore, it would be obvious for one of ordinary skills in the art before the effective filing date of the claimed invention to incorporate the teachings of Chu to acquire information sent by the VR terminal device before acquiring the rendered frame, as the information sent from the device will determine how the frames would be rendered.
The combination of Dai and Chu further teaches determining a distance between the first FOV rendering camera and the second FOV rendering camera according to the pupillary distance information (Chu; [0101] “the viewpoint for the second eye, which is the right eye of the user, is translated to the right of the first eye viewpoint by an interpupillary distance, the distance between the pupils of the user wearing the HMD client device”). This is further taught by Desaulniers, which state that “Without accounting for the IPD offset, a proper parallax effect cannot be achieved. Parallax is very important for differentiating distances to various objects and depth perception” [paragraph 10, lines 1-3]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chu and Desaulniers to account for the pupillary distance such that depth perception and differentiating distances of objects are possible for stereoscopic rendering.
Desaulniers further teaches wherein the device information comprises pupillary distance information and FOV angle information ([paragraph 8, lines 1-2] “The Oculus SDK has a configuration utility where the user can set individual FOV per eye and interpupillary distance”). Desaulniers is analogous to the claimed invention, as both relate to stereoscopic rendering for VR devices. Desaulniers further teaches that the FOV and pupillary information is important for stereoscopic rendering “so that the viewer’s brain is able to correctly fuse two distinct images into one” [paragraph 9, lines 5-6]. Therefore, it would be obvious for one with ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Desaulniers to the combination of Dai and Chu to correctly render the frames for both sides for the VR device to display.
The combination of Dai, Chu and Desaulniers fails to teach the pose matrices comprise a first pose matrix and a second pose matrix, and configuring the first FOV rendering camera according to the first pose matrix and the FOV angle information, and configuring the second FOV rendering camera according to the second pose matrix and the FOV angle information. However, this is known in the art as taught by Chen et al., hereinafter Chen.
Chen teaches the pose matrices comprise a first pose matrix and a second pose matrix ([0081] “corresponding left-eye (or right-eye) view matrix V and the model transformation matrix Mt”)
and configuring the first FOV rendering camera according to the first pose matrix and the FOV angle information, and configuring the second FOV rendering camera according to the second pose matrix and the FOV angle information. ([0081] “By multiplying the left-eye (or right-eye) frustum F by the corresponding left-eye (or right-eye) view matrix V and the model transformation matrix Mt, a 2D projection in the viewing plane of a 3D object can be determined for rendering as a given left-eye (or right-eye) image”. Note: eye frustrum is defined by the field of view of the camera).
Chen is analogous to the claimed invention, as both relate to stereoscopic rendering. Desaulniers further teaches that this is done “because your eyes are also offset from one another, they also have different positions or translations from the position of the viewer” [paragraph 9, lines 1-3]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Chen to the combination of Dai, Chu, and Desaulniers to have pose matrices corresponding to each eye that account for the offset that naturally occurs between the eyes.
In regard to claim 5, the combination of Dai, Chu, Desaulniers, and Chen teaches the method of claim 2, wherein after acquiring a first rendered frame and a second rendered frame, the method further comprises: acquiring pose update information sent by the VR terminal device; (Chen; [0147] “Updates to head-position information 654 may be applied by graphics module 634 to the virtual space and used to render left-eye and right-eye images for display on display 114”)
and updating the pose matrices according to the pose update information (Chen; [0059] “this head-tracking (or eye-tracking) information (e.g., the positions of the eyes of viewer)… may be used… to update the view and frustum matrices and, thus, the rendered left-eye and right-eye images”).
Chen further teaches that “in this way, the rendered images may be optimal from the viewer perspective” [0059]. Therefore, it would be obvious to one of ordinary skill in the art before the effective filing date of the invention to incorporate the teachings of Chen to the combination of Dai, Chu, and Desaulniers for optimal display of rendered images to the user wearing the VR device.
In regards to claim 8, claim 8 recites substantially similar limitations to claim 2. Therefore, the rationale of claim 2 is applied to reject claim 8.
In regards to claim 13, claim 13 recites substantially similar limitations to claim 5. Therefore, the rationale of claim 5 is applied to reject claim 13.
In regards to claim 18, claim 18 recites substantially similar limitations to claim 2. Therefore, the rationale of claim 2 is applied to reject claim 18.
Claims 3, 4, 9 and 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Dai et al. (US 2020/0204783) in view of Chu et al. (US 2020/0092599), Desaulniers (“Stereoscopic Rendering in WebVR”), and Chen et al. (US 2020/0265618), and further in view of Wang et al. (CN 109714585). The citation for Wang et al. is from its respective English translation.
In regards to claim 3, the combination of Dai, Chu, Desaulniers, and Chen teach the method of claim 2, but fail to teach wherein acquiring difference data from the second rendered frame comprises: determining a repetitive area between the first rendered frame and the second rendered frame according to the FOV angle information and the pupillary distance information; determining an area in the second rendered frame other than the repetitive area as a difference area; and acquiring the difference data from the second rendered frame according to the difference area.
However, this is known in the art as taught by Wang et al., hereinafter Wang. Wang teaches wherein acquiring difference data from the second rendered frame comprises: determining a repetitive area between the first rendered frame and the second rendered frame ([0072] “the binocular visible portion is synthesized from the overlapping portion of the left eye and the overlapping portion of the right eye.” Note: the “binocular visible portion” is the repetitive area)
according to the FOV angle information and the pupillary distance information; ([0081] “the status parameters of the display device can be obtained… include interpupillary distance c, monocular field of view α”)
determining an area in the second rendered frame other than the repetitive area as a difference area; and acquiring the difference data from the second rendered frame according to the difference area ([0018] “Based on the binocular overlap ratio, the left eye feature portion and the left eye overlap portion are determined from the left eye image, and the right eye feature portion and the right eye overlap portion are determined from the right eye image”. Note: the “right eye feature portion” is the difference data).
Wang is analogous to the claimed invention, as both relate to stereoscopic rendering by finding the difference area between the left and right eye images. Wang further teaches that “display devices with two screens… needs a high resolution and refresh rate, which requires a large transmission bandwidth for image transmission” [0006], and this embodiment “can reduce the requirements for transmission bandwidth” [0008]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wang to the combination of Dai, Chu, Desaulniers, and Chen to reduce the transmission bandwidth when sending stereoscopic rendered images for dual-screen devices.
In regards to claim 4, the combination of Dai, Chu, Desaulniers, Chen, and Wang teaches the method of claim 3, wherein sending the first rendered frame and the difference data to the VR terminal device such that the VR terminal device restores the second rendered frame according to the first rendered frame and the difference data comprises: adding identification information of the repetitive area to the first rendered frame; (Wang; [0090] “identification information can be sent to the display device”, where “a second identifier for indicating the binocular visible portion”)
and sending the first rendered frame carrying the identification information and the difference data to the VR terminal device, (Wang; [0089] “when sending the first image to the display device, the left-eye feature portion [and] the binocular visual portion… can be sent separately”)
such that the VR terminal device determines the repetitive area according to the identification information, (Wang; [0090] “the display device can identify the left eye feature, the binocular visible part, and the right eye feature from the first image based on the identification information.”)
acquires first repetitive frame data from the first rendered frame according to the repetitive area, (Wang; [0068] “the binocular visual portion can be shared when displaying the left-eye image and the right-eye image”, where “binocular visible portion is synthesized from the left eye overlapping portion and the right eye overlapping portion” [0029])
and restores the second rendered frame according to the first repetitive frame data and the difference data (Wang; [0089] “right-eye feature portion and the binocular visual portion can be used by the display device to display the right-eye image”).
Wang further teaches that with this embodiment, “the total amount of data sent when the first image is sent to the display device is reduced, thereby reducing the requirement for transmission bandwidth” [0097]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wang to the combination of Dai, Chu, Desaulniers, Chen, and Wang to reduce the amount of data that needs to be sent to the device to reduce the bandwidth needed to send data.
In regards to claim 9, claim 9 recites substantially similar limitations to claim 4. Therefore, the rationale of claim 4 is applied to reject claim 9.
In regards to claim 19, claim 19 recites substantially similar limitations to claim 3. Therefore, the rationale of claim 3 is applied to reject claim 19.
In regards to claim 20, claim 20 recites substantially similar limitations to claim 4. Therefore, the rationale of claim 4 is applied to reject claim 20.
Claims 6 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Dai et al. (US 2020/0204783) in view of Chu et al. (US 2020/0092599), and further in view of Fang (US 2015/0003532).
The combination of Dai and Chu teaches the method of claim 1, wherein sending the first rendered frame and the difference data to the VR terminal device comprises: encoding the difference data to obtain a second encoding result; (Dai; [0007] “conducting compressed encoding on the difference information… to obtain compressed difference encoded information”)
and sending the second encoding result to the VR terminal device (Dai; [0010] “receiving difference encoded information… sent by a host of the virtual reality device”).
The combination of Dai and Chu fails to teach acquiring full frame data of the first rendered frame, and encoding the full frame data to obtain a first encoding result; and sending the first encoding result and the second encoding result to the VR terminal device in parallel, such that the VR terminal device decodes the first encoding result to obtain the first rendered frame and decodes the second encoding result to obtain the difference data.
However, this is known in the art by Fang. Fang teaches acquiring full frame data of the first rendered frame, and encoding the full frame data to obtain a first encoding result; ([0050] “encoding the first video image”, where the first video image is “an acquired first video image” [0021])
encoding the difference data to obtain a second encoding result; ([0050] “encoding the… macro blocks”, where “macro blocks [are] the second video image… which are different from those in the first video image” [0054])
and sending the first encoding result and the second encoding result to the VR terminal device in parallel, ([0054] “receiving, from a terminal (video image sender), a first video image of an I-frame video image and macro blocks”. Note: sending in parallel is obvious in the art of stereoscopic rendering, because the data must be sent in parallel such that the left and right images are viewed simultaneously without delay as to ruin the immersion. Therefore, it is not specifically mapped for this rationale.)
such that the VR terminal device decodes the first encoding result to obtain the first rendered frame and decodes the second encoding result to obtain the difference data ([0057] “decoding the first video image and the macro blocks”).
Fang is analogous to the claimed invention, as both relate to image processing for stereoscopy. Fang further teaches that this embodiment allows “sending a 3D video effectively and reliably with a bandwidth” [0005]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Fang to the combination of Dai and Chu to encode the data being sent to the VR device in a reliable and effective fashion.
In regards to claim 14, claim 14 recites substantially similar limitations to claim 6. Therefore, the rationale of claim 6 is applied to reject claim 14.
Allowable Subject Matter
Claims 10, 11, and 12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
In regards to claim 10, the prior art taken singly or in combination do not teach or suggest the limitation “wherein the first repetitive frame data comprises first screen coordinates of each pixel in a screen space coordinate system and further comprises a pixel Red Green Blue (RGB) value corresponding to the first screen coordinates, performing matrix transformation for the first screen coordinates to obtain second screen coordinates; assigning the pixel RGB value to the corresponding second screen coordinates according to a mapping relationship between the first screen coordinates and the second screen coordinates to obtain second repetitive frame data; and combining the second repetitive frame data with the difference data to obtain the second rendered frame.”
Therefore, claim 10 is considered allowable.
Claim 11 contains allowable subject matter because it depends on claim 10 that contains allowable subject matter.
Claim 12 contains allowable subject matter because it depends on claim 11 that contains allowable subject matter.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALICIA HA whose telephone number is (571)272-3601. The examiner can normally be reached Mon-Thurs 8:00 AM - 5:00 PM, and Fri 8:00 AM - 12:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALICIA HA/Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611