DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 5 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 5 recites the limitation "the other sensing data" in line 2. There is insufficient antecedent basis for this limitation in the claim(s). The Examiner notices the manner in which the claim(s) is/are presented, and for purposes for examination will interpret "… image signal processing on the other sensing data" to be "… image signal processing on
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3 and 5-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yu et al., US PGPUB No. 20180011729 A1, hereinafter Yu, in view of Banerjee et al., US PGPUB No. 20240029363 A1, hereinafter Banerjee, and further in view of Gruen et al., US PGPUB No. 20230222741 A1, hereinafter Gruen.
Regarding claim 16, Yu discloses a device (Yu; a computer device [¶ 0050 and ¶ 0058], as illustrated within Fig. 5) comprising:
a first processor configured to generate a first graphic object image in a first frame duration (Yu; the computer device [as addressed above] comprises a 1st processor (corresponding to one or more CPUs (and/or one or more GPUs)) configured to generate a 1st graphic object image (i.e. graphics workload) in a 1st frame duration (i.e. time length) [¶ 0017-0019], as illustrated within Fig. 1; additionally, one operation mode out of multiple operational modes (e.g. native mode and virtualization mode) [id.]);
a second processor configured to perform, based on information, first image signal processing on data of a first image signal in the first frame duration (Yu; the computer device [as addressed above] comprises a 2nd processor (corresponding to one or more GPUs (and/or other CPUs)) configured to perform 1st ISP on data of an implicit 1st image signal (given the throughput of graphics) in the 1st frame duration based on information [¶ 0017-0019], as illustrated within Fig. 1; wherein, performing a (1st) ISP of the 1st image signal corresponds to workload handover between the CPU and GPU [id.]; moreover, distribution and/or scheduling of work [¶ 0016-0017, ¶ 0022, and ¶ 0024]; additionally, virtualization mode associated with one or more VMs and GPU [¶ 0019-0021]; a variety of different possible workloads that may be accelerated using the GPU [¶ 0039]); and
a third processor configured to generate based on the first graphic object image and the sensing data processed by the first image signal processing (Yu; the computer device [as addressed above] comprises a 3rd processor (corresponding to another GPU) configured to generate a virtualization based on the 1st graphic object image and the data processed by the 1st ISP [¶ 0017-0019], as illustrated within Fig. 1).
Yu fails to explicitly disclose an extended reality (XR) device.
Yu further fails to disclose to perform, based on first invisible portion information indicating a first invisible portion of a first real image obscured by the first graphic object image, first image signal processing on sensing data of a first image signal in the first frame duration, wherein the sensing data is different from sensing data corresponding to the first invisible portion; and
to generate a display picture based on the first graphic object image and the sensing data processed by the first image signal processing.
However, Banerjee teaches an extended reality (XR) device (Banerjee; XR device [¶ 0074-0077], as illustrated within Fig. 6);
to perform, based on first invisible portion information indicating a first invisible portion of a first real image obscured by the first graphic object image, first image signal processing on sensing data of a first image signal in the first frame duration (Banerjee; perform 1st ISP (using a GPU) on sensing data of a 1st image signal in the 1st frame duration based on 1st invisible (i.e. occlusion) portion information indicating a 1st invisible (i.e. occlusion) portion of a 1st real image obscured by the 1st graphic object image [¶ 0050-0051 and ¶ 0056]; wherein, sensing data corresponds to image sensor [¶ 0084-0085]; additionally, image warping [¶ 0052-0053 and ¶ 0072]), wherein the sensing data is different from sensing data corresponding to the first invisible portion (Banerjee; the sensing data is different from sensing data corresponding to the 1st invisible (i.e. occlusion) portion [¶ 0084-0086]; wherein, segmentation of image data establishes different categories/classes of sensor data [id.]; even further, additional senor data can be used to modify imaging [¶ 0037-0038]); and
to generate a display picture based on the first graphic object image and the sensing data processed by the first image signal processing (Banerjee; generating a display picture [¶ 0036 and ¶ 0043] based on the 1st graphic object image and the sensing data processed by the 1st ISP (i.e. GPU) [¶ 0050-0052 and ¶ 0069], as illustrated within Figs. 3A-B; wherein, the part of the imaging of the generated display image is based on data from an image sensor [¶ 0084-0086]).
Yu and Banerjee are considered to be analogous art because both pertain to generating and/or managing data in relation with data/media processing, wherein one or more processing units (e.g. parallel processing) are utilized in order to produce optimized computation of data.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu, to incorporate an extended reality (XR) device an extended reality (XR) device; to perform, based on first invisible portion information indicating a first invisible portion of a first real image obscured by the first graphic object image, first image signal processing on sensing data of a first image signal in the first frame duration, wherein the sensing data is different from sensing data corresponding to the first invisible portion; and to generate a display picture based on the first graphic object image and the sensing data processed by the first image signal processing (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Yu as modified by Banerjee fails to explicitly disclose an image signal processing and image signal.
However, Gruen teaches first image signal processing on sensing data of a first image signal in the first frame duration (Gruen; 1st ISP on sensing data of a 1st image signal in the 1st frame duration [¶ 0033 and ¶ 0044-0045], as illustrated within Fig. 3; wherein, sensing/sensor data corresponds to data associated with a camera [¶ 0014-0015 and ¶ 0018]; moreover, ISP hardware pipeline [¶ 0035-0037] in relation with image pixel stream [¶ 0051-0052]).
Yu in view of Banerjee and Gruen are considered to be analogous art because both pertain to generating and/or managing data in relation with data/media processing, wherein one or more processing units are utilized in order to produce optimized imaging.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee, to incorporate first image signal processing on sensing data of a first image signal in the first frame duration (as taught by Gruen), in order to provide improved imaging within a head-mounted display (Gruen; [¶ 0001, ¶ 0010, and ¶ 0013]).
Regarding claim 17, Yu in view of Banerjee and Gruen further discloses the XR device of claim 16, further comprising a fourth processor configured to generate the information in the first frame duration and provide the information to the second processor within a threshold time (Yu; a 4th processor (corresponding to an additional GPU) configured to generate the information in the 1st frame duration and provide the information to the 2nd processor within a threshold time [¶ 0017-0018 and ¶ 0020-0021]; moreover, operating timer(s) that involve a trigger (i.e. threshold) [¶ 0032-0035 and ¶ 0038]).
Banerjee further teaches a processor configured to generate the first invisible portion information in the first frame duration and provide the first invisible portion information to the second processor (Banerjee; a processor configured to generate the 1st invisible (i.e. occlusion) portion information in the 1st frame duration and provide the 1st invisible (i.e. occlusion) portion information to the 2nd processor within a latency [¶ 0050-0052 and ¶ 0056]; additionally, providing techniques for latency [¶ 0028-0030 and ¶ 0069]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate a processor configured to generate the first invisible portion information in the first frame duration and provide the first invisible portion information to the second processor (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 18, Yu in view of Banerjee and Gruen further discloses the XR device of claim 16, further comprising a fourth processor configured to generate the first invisible portion information in a frame duration before the first frame duration and provide the first invisible portion information to the second processor within a threshold time.
further comprising a fourth processor configured to generate the information in a frame duration before the first frame duration and provide the information to the second processor within a threshold time (Yu; a 4th processor (corresponding to an additional GPU) configured to generate the information in the 1st frame duration before the first frame duration and provide the information to the 2nd processor within a threshold time [¶ 0017-0018 and ¶ 0020-0021]; moreover, operating timer(s) that involve a trigger (i.e. threshold) [¶ 0032-0035 and ¶ 0038]).
Banerjee further teaches a processor configured to generate the first invisible portion information in a frame duration before the first frame duration and provide the first invisible portion information to the second processor (Banerjee; a processor configured to generate the 1st invisible (i.e. occlusion) portion information in a frame duration before the 1st frame duration and provide the 1st invisible (i.e. occlusion) portion information to the 2nd processor within a latency [¶ 0050-0052 and ¶ 0056]; wherein, the CPU and GPU distribute processing responsibility in relation with rendering current and previous frames [¶ 0043 and ¶ 0101]; additionally, providing techniques for latency [¶ 0028-0030 and ¶ 0069]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate a processor configured to generate the first invisible portion information in a frame duration before the first frame duration and provide the first invisible portion information to the second processor (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 19, Yu in view of Banerjee and Gruen further discloses the XR device of claim 18, wherein the first invisible portion information is corrected based on movement of the XR device sensed for a certain period from a generation time point of the first invisible portion information (Banerjee; the 1st invisible (i.e. occlusion) portion information is corrected based on movement of the XR device sensed for a certain period from a generation time point of the 1st invisible (i.e. occlusion) portion information [¶ 0100-0101]; moreover, poses of wearable display device[¶ 0047, ¶ 0055-0056, and ¶ 0066]), and wherein, after correction, the first invisible portion information is provided to the second processor (Banerjee; the 1st invisible (i.e. occlusion) portion information is provided to the 2nd processor implicitly after correction (given warping and/or latency techniques) [¶ 0050-0051 and ¶ 0053-0055]; moreover, performing occlusion rendering and performing graphical adjustments [¶ 0028-0030 and ¶ 0056]; and moreover, split rendering [¶ 0063 and ¶ 0066]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate the first invisible portion information is corrected based on movement of the XR device sensed for a certain period from a generation time point of the first invisible portion information, and wherein, after correction, the first invisible portion information is provided to the second processor (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 20, the rejection of claim 20 is addressed within the rejection of claim 16, due to the similarities claim 20 and claim 16 share, therefore refer to the rejection of claim 16 regarding the rejection of claim 20. Although, claim 20 and claim 16 may not be identical, they are considerably comparable or substantially equivalent given their overlapping subject matter. Thus, it is reasonable to reject claim 20 based on the teachings and rational in relation with the prior art within the rejection of claim 16.
Regarding claim 1, the rejection of claim 1 is addressed within the rejection of claim 16, due to the similarities claim 1 and claim 16 share, therefore refer to the rejection of claim 16 regarding the rejection of claim 1. Although, claim 16 and claim 1 may not be identical, they are considerably comparable or substantially equivalent given their overlapping subject matter. However, the subject matter/limitations not addressed by claim 16 is/are addressed below.
Yu discloses generating a second display picture in a second frame duration (Yu; generating a 2nd display picture (i.e. graphics workload) in a 2nd frame duration (i.e. time length) [¶ 0017-0019], as illustrated within Fig. 1).
Yu fails to disclose wherein generating the first display picture comprises generating first invisible portion information by determining a first invisible portion of a first real image obscured by a first graphic object image.
However, Banerjee teaches generating the first display picture comprises generating first invisible portion information by determining a first invisible portion of a first real image obscured by a first graphic object image (Banerjee; generating the 1st display picture comprises generating 1st invisible (i.e. occlusion) portion information by determining a 1st invisible (i.e. occlusion) portion of a 1st real image obscured by a 1st graphic object image [¶ 0050-0051 and ¶ 0056]).
Yu and Banerjee are considered to be analogous art because both pertain to generating and/or managing data in relation with data/media processing, wherein one or more processing units (e.g. parallel processing) are utilized in order to produce optimized computation of data.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu, to incorporate generating the first display picture comprises generating first invisible portion information by determining a first invisible portion of a first real image obscured by a first graphic object image (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
(further refer to the rejection of claim 16)
Regarding claim 2, Yu in view of Banerjee and Gruen further discloses the method of claim 1, wherein the first invisible portion information (Banerjee; the 1st invisible (i.e. occlusion) portion information [as addressed within the parent claim(s)]) comprises:
first sub-information about one or more boundary pixels overlapping with the first graphic object image (Banerjee; the 1st invisible (i.e. occlusion) portion information [as addressed above] comprises (a) 1st sub-information (i.e. depth info) about one or more boundary (i.e. contour) pixels overlapping with the 1st graphic object image [¶ 0029-0030 and ¶ 0055-0056]; wherein, overlapping is based on real and virtual environments [¶ 0050-0051 and ¶ 0064-0065]; wherein, contours are segmented at the pixel level [¶ 0084 and ¶ 0100]; and wherein, adjustments to rendering may include (re-projecting) contours/boundaries [¶ 0092 and ¶ 0108]); and
second sub-information about one or more invisible pixels covered by the first graphic object image (Banerjee; 2nd sub-information about one or more invisible (i.e. occlusion) pixels covered by the 1st graphic object image [¶ 0051 and ¶ 0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate wherein the first invisible portion information comprises: first sub-information about one or more boundary pixels overlapping with the first graphic object image; and second sub-information about one or more invisible pixels covered by the first graphic object image (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 3, Yu in view of Banerjee and Gruen further discloses the method of claim 1, wherein performing the first image signal processing comprises:
determining whether the first invisible portion information is valid in the first frame duration (Banerjee; determining whether the 1st invisible (i.e. occlusion) portion information is valid in the implicit 1st frame duration (given rendering updating and latency) [¶ 0056, ¶ 0092, and ¶ 0100]; wherein, correct/proper blending/overlap of real and virtual environments [¶ 0026 and ¶ 0028-0030] involves occlusion based rendering techniques that further involves identifying portions of a virtual object [¶ 0048 and ¶ 0067-0068]; moreover, (performing) determinations of a correct or improper occlusion (e.g. quality of occlusion handling) in relation with an AR image rendering is known to one of ordinary skill in the art [¶ 0003-0005]; in other words, determinations of invalidity implicitly involves determining validity and vice versa); and
in response to determining that the first invisible portion information is valid, performing the first image signal processing on sensing data other than sensing data corresponding to the first invisible portion information (Banerjee; performing the 1st ISP on sensing data (e.g. captured image data) other than sensing data corresponding to the 1st invisible (i.e. occlusion) portion information in response to determining that the 1st invisible (i.e. occlusion) portion information is valid [¶ 0028-0030 and ¶ 0100]; moreover, re-projection (i.e. performing rendering adjustments) [¶ 0055-0056 and ¶ 0069-0070]; wherein, warping may be applied to virtual objects (and/or real-world objects in mixed reality (MR) use cases) of the rendered image to change the virtual objects (and/or real-world) objects of the rendered image to be displayed in the wearable display device [¶ 0072]; in other words, modifications are performed on the blended image data in response to improper occlusion; wherein, occlusion areas in relation to sensor data can be segmented [¶ 0100-0101]; and wherein, determination of validity [as addressed above]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate performing the first image signal processing comprises: determining whether the first invisible portion information is valid in the first frame duration; and in response to determining that the first invisible portion information is valid, performing the first image signal processing on sensing data other than sensing data corresponding to the first invisible portion information (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 5, Yu in view of Banerjee and Gruen further discloses the method of claim 1, wherein performing the first image signal processing on the other sensing data (Banerjee; performing the 1st image signal processing on other sensing data [¶ 0084-0086]; wherein, segmentation of image data establishes different categories/classes of sensor data [id.]; even further, additional senor data can be used to modify imaging [¶ 0037-0038]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate performing the first image signal processing on the other sensing (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Gruen further teaches converting one or more values of the sensing data corresponding to the first invisible portion information into one or more specific values based on an invisible area filter (Gruen; converting one or more values (using an AR control circuit) of the sensing data (i.e. capture/camera data) corresponding to the 1st invisible portion information [¶ 0024, ¶ 0028-0029, and ¶ 0032-0033] into one or more specific values based on an invisible area filter [¶ 0040-0041], as illustrated within Fig. 3; moreover, ISP of a received camera image pixel stream [¶ 0033 and ¶ 0035-0036] in relation with generating composited imaging of real world and virtual content [¶ 0041-0042]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate converting one or more values of the sensing data corresponding to the first invisible portion information into one or more specific values based on an invisible area filter (as taught by Gruen), in order to provide improved imaging within a head-mounted display (Gruen; [¶ 0001, ¶ 0010, and ¶ 0013]).
Regarding claim 6, Yu in view of Banerjee and Gruen further discloses the method of claim 3, wherein the device (Yu; the device [as addressed within parent claim(s)]) comprises:
a first processor configured to generate the information (Yu; a 1st processor (i.e. CPU or GPU) configured to generate the information [¶ 0017-0019], as illustrated within Fig. 1); and
a second processor configured to perform the first image signal processing (Yu; a 2nd processor (i.e. GPU or CPU) configured to perform the 1st ISP [¶ 0017-0019], as illustrated within Fig. 1; wherein, performing a (1st) ISP of the 1st image signal corresponds to workload handover between the CPU and GPU [id.]; moreover, distribution and/or scheduling of work [¶ 0016-0017, ¶ 0022, and ¶ 0024]; additionally, virtualization mode associated with one or more VMs and GPU [¶ 0019-0021]; a variety of different possible workloads that may be accelerated using the GPU [¶ 0039]), and wherein determining whether the information is valid (Yu; determining whether the information is valid [¶ 0032-0034]; moreover, timer expires then a deadlock timer expires [¶ 0038], and determining error in relation with determining/validating completion [¶ 0035, ¶ 0037, and ¶ 0045-0047]) comprises:
determining whether the information from the first processor can be provided to the second processor within a threshold time (Yu; determining whether the information from the 1st processor (i.e. CPU or GPU) can be provided to the second processor (i.e. GPU or CPU) within a threshold time [¶ 0016 and ¶ 0018]).
Banerjee further teaches wherein the XR device (Banerjee; the XR device [as addressed within parent claim(s)]);
to generate the first invisible portion information (Banerjee; generating the 1st invisible (i.e. occlusion) portion information [¶ 0050-0051 and ¶ 0056]); and
to perform the first image signal processing (Banerjee; performing the 1st ISP (using a GPU) [¶ 0050-0051 and ¶ 0056]), and wherein determining whether the first invisible portion information is valid (Banerjee; determining whether the 1st invisible (i.e. occlusion) portion information is valid [¶ 0056, ¶ 0092, and ¶ 0100]; moreover, occlusion based rendering involves identifying portions of a virtual object 0048 and ¶ 0067-0068] in relation with correct/proper blending/overlap of real and virtual environments [¶ 0026 and ¶ 0028-0030]; such that, determinations of invalidity implicitly involves determining validity and vice versa) comprises:
determining whether the first invisible portion information from the first processor can be provided to the second processor within a threshold time (Banerjee; determining whether the 1st invisible (i.e. occlusion) portion information from the 1st processor can be implicitly provided to the 2nd processor (given distribution of tasks) within a threshold time [¶ 0050-0052 and ¶ 0066], as illustrated within Fig. 4; wherein, Fig. 4 illustrates, multiple timelines for which rendering tasks are to be performed by one or more components; and wherein, tasks are performed within a multi-processor system [¶ 0034, ¶ 0043, and ¶ 0045], such that a CPU and/or GPU are configured to perform graphics processing [¶ 0047-0048]; additionally, rendering (associated with time warping) to be performed by a GPU according to a determined time interval [¶ 0052]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate wherein the XR device; to generate the first invisible portion information; and to perform the first image signal processing, and wherein determining whether the first invisible portion information is valid comprises: determining whether the first invisible portion information from the first processor can be provided to the second processor within a threshold time (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 7, Yu in view of Banerjee and Gruen further discloses the method of claim 6, wherein the first image signal processing starts later than a start time point of the first frame duration by at least the threshold time (Banerjee; the 1st ISP starts later than a start time point of the implicit 1st frame duration (given rendering updating and latency) by at least the threshold time [¶ 0055-0056 and ¶ 0066], as illustrated within Fig. 4; moreover, image processing associated with late stage occlusion [¶ 0069-0070]; and moreover, re-projection and occlusion techniques in relation with latency processing [¶ 0071-0072 and ¶ 0092]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate wherein the first image signal processing starts later than a start time point of the first frame duration by at least the threshold time (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 8, Yu in view of Banerjee and Gruen further discloses the method of claim 6, wherein the XR device further comprises a third processor configured to composite the processed first sensing data with the first graphic object image (Yu; the device further comprises a 3rd processor (corresponding to another GPU) configured to process 1st sensing data with the 1st graphic object image [¶ 0017-0019]; wherein, Fig. 1 illustrates, a distribution of workloads among multiple processing units), and the threshold time is set based on at least one of the performance of the second processor, a performance of the third processor, or a length of the first frame duration (Yu; the threshold time is set based on (at least) the performance of the second processor [¶ 0032-0034]; in other words, VM timer and/or deadlock timer are based on the performance of the VM and/or GPU respectively).
Banerjee further teaches the XR device (Banerjee; the XR device [as addressed within the parent claim(s)]); and
to composite the processed first sensing data with the first graphic object image (Banerjee; compositing the processed 1st sensing data with the 1st graphic object image [¶ 0051-0051 and ¶ 0100]), and the threshold time is set based on at least one of the performance of the second processor, a performance of the third processor, or a length of the first frame duration (Banerjee; the threshold time is set based on (at least) a length of the implicit 1st frame duration (given rendering updating and latency) [¶ 0066]; moreover, latency handling [¶ 0028-0030 and ¶ 0052]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate the XR device; and to composite the processed first sensing data with the first graphic object image, and the threshold time is set based on at least one of the performance of the second processor, a performance of the third processor, or a length of the first frame duration (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 9, Yu in view of Banerjee and Gruen further discloses the method of claim 3, wherein performing the first image signal processing further comprises performing the first image signal processing on the first sensing data in response to determining that the first invisible portion information is not valid (Banerjee; performing the 1st ISP further comprises performing the 1st ISP on the 1st sensing data [as addressed within the parent claim(s)] in response to determining that the 1st invisible (i.e. occlusion) portion information is not valid (i.e. improper) [¶ 0070-0072 and ¶ 0100]; wherein, re-projection is a responsive mechanism [¶ 0055-0056 and ¶ 0092]; moreover, determination of validity or invalidity [as further addressed within the parent claim(s)]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate performing the first image signal processing further comprises performing the first image signal processing on the first sensing data in response to determining that the first invisible portion information is not valid (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 10, Yu in view of Banerjee and Gruen further discloses the method of claim 3, wherein the first invisible portion information that is not valid in the first frame duration is used in generating the second display picture (Banerjee; the 1st invisible (i.e. occlusion) portion information that is not valid in the implicit 1st frame duration (given rendering updating and latency) is used in generating the 2nd (i.e. updated) display picture [¶ 0100-0101]; wherein, re-projection is a mechanism to update imaging [¶ 0028-0030, ¶ 0070-0072, and ¶ 0092]; such that, re-projection involves rendering adjustment [¶ 0055-0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate the first invisible portion information that is not valid in the first frame duration is used in generating the second display picture (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 11, Yu in view of Banerjee and Gruen further discloses the method of claim 3, wherein the first invisible portion information that is not valid in the first frame duration (Banerjee; the 1st invisible (i.e. occlusion) portion information that is not valid in the implicitly 1st frame duration (given rendering updating and latency) [as addressed within the parent claim(s)]) is corrected based on at least one of:
information related to the first graphic object image, or motion information of the XR device (Banerjee; the 1st invisible (i.e. occlusion) portion information that is not valid in the implicit 1st frame duration (given rendering updating and latency) [as addressed above]) is corrected based (at least) on motion information of the XR device [¶ 0100-0101]; wherein, performing time warping techniques based (in part) on updated position information for wearable display device [¶ 0052-0053]; additionally, occlusion rendering [¶ 0050-0051 and ¶ 0055-0056] and late stage occlusion rendering [¶ 0069-0070]), and wherein the corrected first invisible portion information is used in generating the second display picture (Banerjee; wherein the corrected 1st invisible (i.e. occlusion) portion information is used in generating the 2nd (i.e. updated) display picture [¶ 0100-0101]; wherein, re-projection is a mechanism to update imaging [¶ 0070-0072 and ¶ 0092]; such that, re-projection involves rendering adjustment [¶ 0055-0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate wherein the first invisible portion information that is not valid in the first frame duration is corrected based on at least one of: information related to the first graphic object image, or motion information of the XR device, and wherein the corrected first invisible portion information is used in generating the second display picture (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 12, Yu in view of Banerjee and Gruen further discloses the method of claim 11, wherein a number of invisible pixels in the corrected first invisible portion information is less than a number of invisible pixels in the first invisible portion information (Banerjee; a number of invisible (i.e. occluded) pixels in the corrected 1st invisible (i.e. occlusion) portion information is less than a number of invisible (i.e. occluded) pixels in the 1st invisible (i.e. occlusion) portion information [¶ 0029-0030 and ¶ 0100-0101]; moreover, updated in relation with a smaller depth pixel [id.]; additionally, reduced depth buffer size [¶ 0080]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate wherein a number of invisible pixels in the corrected first invisible portion information is less than a number of invisible pixels in the first invisible portion information (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 13, Yu in view of Banerjee and Gruen further discloses the method of claim 1, wherein the first invisible portion information (Banerjee; the 1st invisible (i.e. occlusion) portion information [as addressed within the parent claim(s)]) is generated based on at least one of:
object coordinate information for the first graphic object image, object motion vector information for the first graphic object image, or motion information of the XR device (Banerjee; the 1st invisible (i.e. occlusion) portion information [as addressed above] is generated based (at least) on motion information of the XR device [¶ 0100-0101]; wherein, performing time warping techniques based (in part) on updated position information for wearable display device [¶ 0052-0053]; additionally, occlusion rendering [¶ 0050-0051 and ¶ 0055-0056] and late stage occlusion rendering [¶ 0069-0070]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate the first invisible portion information is generated based on at least one of: object coordinate information for the first graphic object image, object motion vector information for the first graphic object image, or motion information of the XR device (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Regarding claim 14, Yu in view of Banerjee and Gruen further discloses the method of claim 1, further comprising:
identifying characteristics of an application running on the device (Yu; identifying characteristics (i.e. command ring info, head and tail info) of an application running on the device [¶ 0023-0025]; additionally, identifying characteristics correspond to determining latency [¶ 0027-0028, ¶ 0064, and ¶ 0066]); and
setting a parameter for the first image signal processing based on the identified characteristics (Yu; setting a parameter for the 1st ISP based on the identified characteristics (i.e. command ring info, head and tail info) [¶ 0026-0028]; wherein, a command ring comprises head and tail registers [id.], in relation with tracking and progressing [¶ 0030-0031]; additionally, determining a time interval (i.e. parameter) due to latency in processing [¶ 0069-0071]).
Gruen further teaches identifying characteristics of an application running on the XR device (Gruen; identifying characteristics (i.e. buffering) of an application running on the XR device [¶ 0033-0035]; additionally, identifying characteristics also corresponds to determining poses, frame rates, and/or processing capabilities [¶ 0028-0029 and ¶ 0031]); and
setting a parameter for the first image signal processing based on the identified characteristics (Gruen; setting a parameter (i.e. group/grouping) for the 1st ISP based on the identified characteristics (i.e. buffering) [¶ 0033-0034]; wherein, a pixel buffer can be utilized to determine coordinates of pixels on a display [¶ 0035 and ¶ 0037-0038]; additionally, color parameters for the ISP based on latency [¶ 0036-0038]; moreover, reducing latency [¶ 0011-0013]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate identifying characteristics of an application running on the XR device; and setting a parameter for the first image signal processing based on the identified characteristics (as taught by Gruen), in order to provide improved imaging within a head-mounted display (Gruen; [¶ 0001, ¶ 0010, and ¶ 0013]).
Regarding claim 15, Yu in view of Banerjee and Gruen further discloses the method of claim 14, wherein the parameter comprises at least one of:
a first parameter related to information that forms a basis for generating the first invisible portion information; or
a second parameter related to information that forms a basis for determining whether the information is valid in the first frame duration (Yu; the parameter [as addressed within the parent claim(s)] comprises (at least) a 2nd parameter (i.e. tail) related to information that forms a basis for determining whether the information is valid in the 1st frame duration [¶ 0016 and ¶ 0027-0028]).
Banerjee further teaches a second parameter related to information that forms a basis for determining whether the first invisible portion information is valid in the first frame duration (Banerjee; a 2nd parameter (i.e. latency) related to information that forms a basis for determining whether the 1st invisible (i.e. occlusion) portion information is valid in the implicit 1st frame duration (given rendering updating and latency) [¶ 0070-0071 and ¶ 0100]; moreover, determination of validity or invalidity [as further addressed within the parent claim(s)]; additionally, late stage occlusion rendering [¶ 0069 and ¶ 0072]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate performing the first image signal processing further comprises performing the first image signal processing on the first sensing data in response to determining that the first invisible portion information is not valid (as taught by Banerjee), in order to provide an improved blending between virtual and real environments that allow for realistic imaging (Banerjee; [¶ 0002-0003, ¶ 0005, and ¶ 0007]).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yu in view of Banerjee and Gruen as applied to claim(s) 3 above, and further in view of Wellington et al., US PGPUB No. 20170262709 A1, hereinafter Wellington.
Regarding claim 4, Yu in view of Banerjee and Gruen further discloses the method of claim 3, wherein performing the first image signal processing on the other sensing data (Banerjee; performing the 1st ISP on the other sensing data [as addressed within the parent claim(s)]) comprises:
generating a bypass signal to bypass the sensing data corresponding to the first invisible portion information in the first image signal processing (Banerjee; the sensing data [¶ 0084-0086] corresponding to the 1st invisible (i.e. occlusion) portion information in the 1st ISP [¶ 0050-0052 and ¶ 0069], as illustrated within Figs. 3A-B).
Yu as modified by Banerjee and Gruen fails to disclose generating a bypass signal to bypass the sensing data corresponding to the first invisible portion information.
However, Wellington teaches generating a bypass signal to bypass the sensing data corresponding to the first invisible portion information in the first image signal processing (Wellington; generating a bypass signal to bypass the sensing data corresponding to the 1st invisible (i.e. occlusion) portion information in the 1st ISP [¶ 0070]).
Yu in view of Banerjee and Gruen and Wellington are considered to be analogous art because they pertain to generating and/or managing data in relation with data/media processing, wherein one or more processing units are utilized in order to produce optimized imaging.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing of the claimed invention was made to modify Yu as modified by Banerjee and Gruen, to incorporate generating a bypass signal to bypass the sensing data corresponding to the first invisible portion information in the first image signal processing (as taught by Wellington), in order provide improved efficient/timely processing of image analysis within real interactive environments (Wellington; [¶ 0016 and ¶ 0023]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Refer to PTO-892, Notice of Reference Cited for a listing of analogous art.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Charles Lloyd Beard whose telephone number is (571)272-5735. The examiner can normally be reached Monday - Friday, 8:00 AM - 5: 00 PM, alternate Fridays EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
CHARLES LLOYD. BEARD
Primary Examiner
Art Unit 2611
/CHARLES L BEARD/Primary Examiner, Art Unit 2611
Prosecution Insights