DETAILED ACTION
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 - 30 are rejected under 35 U.S.C. 103 as being unpatentable over Eiden et al. (Publication: US 2021/0397251 A1) in view of Margolis et al. (Publication: US 2017/0213388 A1).
Regarding claim 1, see rejection on clam 16.
Regarding claim 2, see rejection on clam 17.
Regarding claim 3, see rejection on clam 18.
Regarding claim 4, see rejection on clam 19.
Regarding claim 5, see rejection on clam 20.
Regarding claim 6, see rejection on clam 21.
Regarding claim 7, see rejection on clam 22.
Regarding claim 8, see rejection on clam 23.
Regarding claim 9, see rejection on clam 24.
Regarding claim 10, see rejection on clam 25.
Regarding claim 11, see rejection on clam 26.
Regarding claim 12, see rejection on clam 27.
Regarding claim 13, see rejection on clam 28.
Regarding claim 14, see rejection on clam 29.
Regarding claim 15, see rejection on clam 30.
Regarding claim 16, Eiden discloses an apparatus for generating one or more frames, comprising ([0025] - the display system is implemented as a display apparatus and an external computing device communicably coupled to the display apparatus. ):
at least one memory; and at least one processor coupled to the at least one memory and configured to ([0025] - the display system is implemented as a display apparatus and an external computing device communicably coupled to the display apparatus.
[0090] Referring to FIG. 2, illustrated is a block diagram of architecture of a display system 200. The display system 200 comprises eye-tracking means 202, head-tracking means 204, a first image renderer 206, a second image renderer 208, an optical combiner 210 and at least one processor depicted as a processor 212. The processor 212 is communicably coupled to the eye-tracking means 202, the head-tracking means 204, the first image renderer 206 and the second image renderer 208. It is know that computing devices comprises memory. ):
obtain a first portion of the frame based on information corresponding to a region of interest (ROI), the first portion having a first resolution ( [0039] when the user's gaze is fixated on an object of interest in the extended-reality scene, it is beneficial to produce a high-resolution image of the object of interest (namely, the second image) at and around the fovea of the user's eye.
[0062] - a. identify, based on the gaze location of the user, an object of interest in the extended-reality scene;
b. determine whether the object of interest is located in a central region or a peripheral region of a field of view of the extended-reality scene “”obtain a first portion. );
obtain a second portion of the frame, the second portion having a second resolution that is lower than the first resolution ([0042] receive the resolution information. the resolution of the projection of the rendered second image is higher than the resolution of the projection of the rendered first image that is the resolution of the first image is lower than the resolution of second image. a resolution of the second image is higher than a resolution of the first image. In other implementations, the resolution of the second image is same as the resolution of the first image. However, in such implementations, the display apparatus further comprises at least one optical element that is arranged to de-magnify the projection of the second image, thereby increasing its apparent resolution. ); and
output the first portion of the frame and the second portion of the frame ([0041], the projection of the rendered first image is optically combined (by an optical combiner) with the projection of the rendered second image to create the aforesaid extended reality-scene. The projection of the rendered first image and the projection of the rendered second image are superimposed to present the extended-reality scene to the user).
Eiden does not however Margolis discloses
obtain, from an image sensor, sensor data for a frame associated with a scene ([0044] - HMD 200 may include two or more front facing cameras (e.g., one on each temple) in order to obtain depth from stereo information associated with the field of view, scene, captured by the front facing cameras.).;
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Eiden with obtain, from an image sensor, sensor data for a frame associated with a scene as taught by Margolis. The motivation for doing is to realistically integrate objects into an AR environment..
Regarding claim 17, Eden in view of Margolis disclose all the limitation of claim 16 including image sensor.
Eiden discloses wherein the at least one processor is configured to obtain the first portion of the frame and the second portion of the frame from the [[image sensor]] ([0041], the projection of the rendered first image is optically combined (by an optical combiner) with the projection of the rendered second image to create the aforesaid extended reality-scene. In other words, the projection of the rendered first image and the projection of the rendered second image are superimposed to present the extended-reality scene to the user, “logical channel of an interface”.
[0042] Furthermore, the resolution of the projection of the rendered second image is higher than the resolution of the projection of the rendered first image. a resolution of the second image is higher than a resolution of the first image “logical channel of an interface”.).
Regarding claim 18, Eden in view of Margolis disclose all the limitation of claim 17 including image sensor.
Eiden discloses receive a mask associated with the scene, wherein the mask includes the information corresponding to the ROI associated with a frame ([0069] When generating the given first image, the at least one processor is configured to mask a region of the given first image that corresponds to the region of interest of the given input image, wherein a projection of the masked region of the rendered first image is to substantially overlap with the projection of the rendered second image.).
Margolis discloses associated with a previous frame ([0085] - The motion vectors may be calculated by comparing the rendered image with a previous image rendered by the rendering module, such as the previous frame immediately preceding the current frame of the rendered image. The motion vectors are generated by comparing features or pixels within each block between the two frames to statistically or otherwise estimate motion at the block between the two frames. In this example, the motion vectors are illustrated by arrows.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Eiden in view of Margolis with associated with a previous frame as taught by Margolis. The motivation for doing is to realistically integrate objects into an AR environment..
Regarding claim 19, Eden in view of Margolis disclose all the limitation of claim 18 including image sensor.
Eiden discloses to determine the mask based on at least one of gaze information of a user, a predicted gaze of the user, an object detected in the scene, a depth map generated for the scene, and a saliency map of the scene (
[0038] It will be appreciated that a change in the gaze location (namely, a shift in the user's gaze) generally indicates a realignment of a line of sight in order to bring an image of a moving object of interest or a new object of interest at and around a fovea of the user's eye. The fovea is a portion of the eye that is typically located at a center of a retina of the eye, where a receptor density and hence a visual resolution are the highest. Thus, it is generally known that the fovea is the portion having clearest vision.
[0069] when generating the given first image, the at least one processor is configured to mask a region of the given first image that corresponds to the region of interest of the given input image, wherein a projection of the masked region of the rendered first image is to substantially overlap with the projection of the rendered second image.).
Regarding claim 20, Eden in view of Margolis disclose all the limitation of claim 17 including image sensor.
Eiden discloses to generate an output frame at least in part by combining the first portion of the frame and the second portion of the frame ([0007] - d. when the user's gaze has been fixated for at least the predefined time period, processing a given input image to generate and render a given first image and a given second image substantially simultaneously via a first image renderer and a second image renderer respectively, wherein a projection of the rendered first image and a projection of the rendered second image are combined optically by an optical combiner to create an extended-reality scene, and wherein the projection of the rendered second image substantially overlaps with a portion of the projection of the rendered first image, further wherein a resolution of the projection of the rendered second image is higher than a resolution of the projection of the rendered first image;).
Regarding claim 21, Eden in view of Margolis disclose all the limitation of claim 17 including image sensor.
Eiden discloses using an image signal processor, the first portion of the frame based on first one or more parameters to improve visual fidelity of the first portion and refraining from processing of the second portion of the frame ([0007] - d. when the user's gaze has been fixated for at least the predefined time period, processing a given input image to generate and render a given first image and a given second image substantially simultaneously via a first image renderer and a second image renderer respectively, wherein a projection of the rendered first image and a projection of the rendered second image are combined optically by an optical combiner to create an extended-reality scene, and wherein the projection of the rendered second image substantially overlaps with a portion of the projection of the rendered first image, further wherein a resolution of the projection of the rendered second image is higher than a resolution of the projection of the rendered first image, “fidelity”.
[0025] – a processor to perform the method above.).
Regarding claim 22, Eden in view of Margolis disclose all the limitation of claim 17 including image sensor.
Eiden discloses combine a plurality of pixels of the sensor data such that the second portion of the frame has the second resolution ([0007] - d. when the user's gaze has been fixated for at least the predefined time period, processing a given input image to generate and render a given first image and a given second image substantially simultaneously via a first image renderer and a second image renderer respectively, wherein a projection of the rendered first image and a projection of the rendered second image are combined optically by an optical combiner to create an extended-reality scene, and wherein the projection of the rendered second image substantially overlaps with a portion of the projection of the rendered first image, further wherein a resolution of the projection of the rendered second image is higher than a resolution of the projection of the rendered first image.).
Regarding claim 23, Eden in view of Margolis disclose all the limitation of claim 17 including image sensor.
Eiden discloses output the first portion of the frame using a first logical channel of an interface between the [[image sensor]] and an image signal processor; and output the second portion of the frame using a second logical channel of the interface ([0041], the projection of the rendered first image is optically combined (by an optical combiner) with the projection of the rendered second image to create the aforesaid extended reality-scene. In other words, the projection of the rendered first image and the projection of the rendered second image are superimposed to present the extended-reality scene to the user, “logical channel of an interface”.
[0042] Furthermore, the resolution of the projection of the rendered second image is higher than the resolution of the projection of the rendered first image. a resolution of the second image is higher than a resolution of the first image “logical channel of an interface”.).
Regarding claim 24, Eden in view of Margolis disclose all the limitation of claim 16 including image sensor.
Eiden discloses to output the first portion of the frame and the second portion of the frame ([0007] - d. when the user's gaze has been fixated for at least the predefined time period, processing a given input image to generate and render a given first image and a given second image substantially simultaneously via a first image renderer and a second image renderer respectively, wherein a projection of the rendered first image and a projection of the rendered second image are combined optically by an optical combiner to create an extended-reality scene, and wherein the projection of the rendered second image substantially overlaps with a portion of the projection of the rendered first image, further wherein a resolution of the projection of the rendered second image is higher than a resolution of the projection of the rendered first image;).
Regarding claim 25 , Eden in view of Margolis disclose all the limitation of claim 24 including image sensor.
Eiden discloses the ROI associated with the scene based on motion information from at least one motion sensor that identifies motion associated with a device including the [[image sensor]] ([0038] It will be appreciated that a change in the gaze location (namely, a shift in the user's gaze) generally indicates a realignment of a line of sight in order to bring an image of a moving object of interest or a new object of interest at and around a fovea of the user's eye. The fovea is a portion of the eye that is typically located at a center of a retina of the eye, where a receptor density and hence a visual resolution are the highest. Thus, it is generally known that the fovea is the portion having clearest vision, shift is “motion”.
[0035] - The eye-tracking means could be implemented as contact lenses with sensors, cameras monitoring a position of a pupil of the user's eye.).
Regarding claim 26, Eden in view of Margolis disclose all the limitation of claim 24 including image sensor.
Eiden discloses determine the ROI based on at least one of gaze information of a user, a predicted gaze of the user, an object detected in the scene, a depth map generated for the scene, and a saliency map of the scene ([0038] It will be appreciated that a change in the gaze location (namely, a shift in the user's gaze) generally indicates a realignment of a line of sight in order to bring an image of a moving object of interest or a new object of interest at and around a fovea of the user's eye. The fovea is a portion of the eye that is typically located at a center of a retina of the eye, where a receptor density and hence a visual resolution are the highest. Thus, it is generally known that the fovea is the portion having clearest vision.).
Regarding claim 27, Eden in view of Margolis disclose all the limitation of claim 26 including image sensor.
Eiden discloses obtain motion information from at least one motion sensor that identifies motion associated with a device including the [[image sensor]] ([0050] - b. identify, based on the gaze location of the user, at least one of the pluralities of pixel segments at which the user is gazing, wherein the at least one of the plurality of pixel segments represents the object of interest. a. determine a region of interest of the given input image based on at least one of: the gaze directions of the user's eyes, scene information of the extended-reality scene; [0048] In such cases, the user's gaze closely follows the object of interest and shifts in a steady and smooth manner, so as to form an image of the object of interest on the fovea of the user's eye. Such an eye movement is known as “smooth-pursuit eye movement”, and is well-known in the art. Thus, smooth-pursuit eye movements allow clear vision of a moving object by holding the image steady on the fovea. In all the above-mentioned cases, the eye is fixated (namely, the user is looking) at the object of interest "motion".
[0035] - The eye-tracking means could be implemented as contact lenses with sensors, cameras monitoring a position of a pupil of the user's eye.); and
modify the ROI based on the motion information ([0078] - a. determining a region of interest of the given input image based on at least one of: the gaze directions of the user's eyes, scene information of the extended-reality scene; b. generating the given second image from the region of interest of the given input image; and c. generating the given first image from at least a region of the given input image that includes and surrounds the region of interest of the given input image.).
Regarding claim 28, Eden in view of Margolis disclose all the limitation of claim 26 including image sensor.
Eiden discloses obtain motion information from at least one motion sensor that identifies motion associated with eyes of the user; and modify the ROI based on the motion information (([0050] - b. identify, based on the gaze location of the user, at least one of the pluralities of pixel segments at which the user is gazing, wherein the at least one of the plurality of pixel segments represents the object of interest. a. determine a region of interest of the given input image based on at least one of: the gaze directions of the user's eyes, scene information of the extended-reality scene; [0048] In such cases, the user's gaze closely follows the object of interest and shifts in a steady and smooth manner, so as to form an image of the object of interest on the fovea of the user's eye. Such an eye movement is known as “smooth-pursuit eye movement”, and is well-known in the art. Thus, smooth-pursuit eye movements allow clear vision of a moving object by holding the image steady on the fovea. In all the above-mentioned cases, the eye is fixated (namely, the user is looking) at the object of interest "motion".
[0078] - a. determining a region of interest of the given input image based on at least one of: the gaze directions of the user's eyes, scene information of the extended-reality scene; b. generating the given second image from the region of interest of the given input image; and c. generating the given first image from at least a region of the given input image that includes and surrounds the region of interest of the given input image.
[0035] - The eye-tracking means could be implemented as contact lenses with sensors, cameras monitoring a position of a pupil of the user's eye.
).
Regarding claim 29, Eden in view of Margolis disclose all the limitation of claim 28 including image sensor.
Margolis discloses increase a size of the ROI in a direction of the motion ([0085] - The length of the arrow corresponds with the size of the motion vector. A longer arrow indicates a larger motion vector representing a larger movement at the block between frames. A smaller arrow indicates a smaller motion vector representing a smaller movement at the block between frames.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Eiden in view of Margolis with increase a size of the ROI in a direction of the motion as taught by Margolis. The motivation for doing is to realistically integrate objects into an AR environment..
Regarding claim 30, Eden in view of Margolis disclose all the limitation of claim 24 including image sensor.
Eiden discloses obtain motion information from at least one motion sensor that identifies motion associated with eyes of a user (
[0050] - b. identify, based on the gaze location of the user, at least one of the pluralities of pixel segments at which the user is gazing, wherein the at least one of the plurality of pixel segments represents the object of interest. a. determine a region of interest of the given input image based on at least one of: the gaze directions of the user's eyes, scene information of the extended-reality scene; [0048] In such cases, the user's gaze closely follows the object of interest and shifts in a steady and smooth manner, so as to form an image of the object of interest on the fovea of the user's eye. Such an eye movement is known as “smooth-pursuit eye movement”, and is well-known in the art. Thus, smooth-pursuit eye movements allow clear vision of a moving object by holding the image steady on the fovea. In all the above-mentioned cases, the eye is fixated (namely, the user is looking) at the object of interest "motion".
[0035] - The eye-tracking means could be implemented as contact lenses with sensors, cameras monitoring a position of a pupil of the user's eye.); and
modify the ROI based on the motion information ([0078] - a. determining a region of interest of the given input image based on at least one of: the gaze directions of the user's eyes, scene information of the extended-reality scene; b. generating the given second image from the region of interest of the given input image; and c. generating the given first image from at least a region of the given input image that includes and surrounds the region of interest of the given input image. ).
Response to Arguments
Claim Rejection Under 35 U.S.C. 103
Examiner suggests to amend a specific element in the claim that when reading a claim in light of the invention, it directs to a unique technology. The examiner can be reached at 571-270-0724 for further discussion.
Applicant asserts “Applicant submits that the combination of Eiden and Margolis fails to describe "capturing, using an image sensor, sensor data for a frame associated with a scene; generating a first portion of the frame based on information corresponding to a region of interest (ROI), the first portion having a first resolution; generating a second portion of the frame, the second portion having a second resolution that is lower than the first resolution; and outputting the first portion of the frame and the second portion of the frame" as recited in claim 1.”
Examiner disagrees.
During patent examination, the pending claims must be given their broadest reasonable interpretation consistent with the specification. See MPEP § 2111. Further, although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). See also MPEP § 2145(VI).
It is the Eiden in view of Margolis the disclose the limitations above.
Margolis discloses capturing, using an image sensor, sensor data for a frame associated with a scene ([0044] - HMD 200 may include two or more front facing cameras (e.g., one on each temple) in order to obtain depth from stereo information associated with the field of view, scene, captured by the front facing cameras.).
Eiden discloses [0039] when the user's gaze is fixated on an object of interest in the extended-reality scene, it is beneficial to produce a high-resolution image of the object of interest (namely, the second image) at and around the fovea of the user's eye. [0062] - a. identify, based on the gaze location of the user, an object of interest in the extended-reality scene;
b. determine whether the object of interest is located in a central region or a peripheral region of a field of view of the extended-reality scene “”obtain a first portion.
[0042] receive the resolution information. the resolution of the projection of the rendered second image is higher than the resolution of the projection of the rendered first image that is the resolution of the first image is lower than the resolution of second image. a resolution of the second image is higher than a resolution of the first image. In other implementations, the resolution of the second image is same as the resolution of the first image. However, in such implementations, the display apparatus further comprises at least one optical element that is arranged to de-magnify the projection of the second image, thereby increasing its apparent resolution.
[0041], the projection of the rendered first image is optically combined (by an optical combiner) with the projection of the rendered second image to create the aforesaid extended reality-scene. The projection of the rendered first image and the projection of the rendered second image are superimposed to present the extended-reality scene to the user.
Regarding claims 2 – 15, and 17 – 30, the Applicant asserts that they are not obvious over based on their dependency from independent claims 1, and 16 respectively. The examiner cannot concur with the Applicant respectfully from same reason noted in the examiner’s response to argument asserted from claims 1, and 16 respectively.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Wu whose telephone number is (571) 270-0724. The examiner can normally be reached on Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ming Wu/
Primary Examiner, Art Unit 2618