Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on January 26th, 2024 has been considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-10, 12, 16, 20-22, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Haine (US 2021/0208674) in view of Shtukater (US 2019/0179165).
Regarding claim 1, Haine discloses an eye-mounted device (Figs. 1-10, element 102) comprising:
a display (120) that projects pixels onto a retina of a user's eye ([0045], “femtoprojector 120 may be implemented as a miniature video projector that comprises an image source”);
a memory storing an application ([0050], “computer memory”), and a processing device coupled to the memory to execute the application ([0076], “a processing unit 1020”), wherein executing the application ([0065], “A virtual tool 550C positioned on the tool ring 530 is selected by a user”, examiner interprets tool to correspond to application) causes the processing device to populate a canvas with images of rendered graphic objects used by the application ([0050], “Objects 406A and partial objects 406B in FIG. 4C that do not fall within visible section 310 are not displayed on the retina and, thus remain invisible to the eye until being recalled from computer memory (or derived from stored information) and included within SoE 304 by the image projector”, examiner interprets the objects not displayed to be formed on a canvas); and
a real-time graphics module configured to, repeatedly and in real-time, transfer the images of the rendered graphic objects from the canvas to the display ([0050], “the image projector that renders the recalled objects onto the retina, in response to the user turning their eye in the direction of those objects … the image projector effectively updates the SoE 304 relative to eye movements of a user by loading a corresponding portion of the virtual image and updating what is projected onto the eye”, examiner interprets the graphics module to correspond to the image projector that updates the SoE in real time).
Haine does not specifically disclose wherein the display, memory, processing device and real-time graphics module are contained within the eye-mounted device.
However Shtukater, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the display (Fig. 9, element 903), memory ([0185], “Communication module 906 may include onboard memory”), processing device ([0187], “processor module 910”) and real-time graphics module ([0191], “the processor module 910 may output visual information to the embedded display device 903”) are contained within the eye-mounted device (as shown in fig. 9, display, memory, processor and graphics module are all contained in lens 901).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine with the wherein the display, memory, processing device and real-time graphics module are contained within the eye-mounted device as taught by Shtukater, for the purpose of creating a better augmented reality experience ([0191]).
Regarding claim 2, modified Haine teaches as is set forth in claim 1 rejection above and Haine further discloses wherein the real-time graphics module is further configured to receive an eye angle of the user's eye ([0031], “Data derived from such eye movements may include timing and sequences of saccadic movements, eye direction (e.g., eye angle”); determine, based on the eye angle, which graphic objects fall within a display window of the display (as shown in Fig. 4C, based on angle of view only certain objects are displayed with bold lines); and transfer the images for those graphic objects from the canvas to the display ([0050], “the image projector effectively updates the SoE 304 relative to eye movements of a user by loading a corresponding portion of the virtual image and updating what is projected onto the eye.”).
Regarding claim 3, modified Haine teaches as is set forth in claim 2 rejection above and Haine further discloses further comprising:
one or more accelerometers, a gyroscope and a magnetometer ([0027], “motion sensors 125, such as accelerometers, magnetometers, and gyroscopes”); and
an eye tracking unit that determines the eye angle based on measurements from the one or more accelerometers, the gyroscope and the magnetometer ([0027], “that may be used for eye-tracking functionality”).
Regarding claim 5, modified Haine teaches as is set forth in claim 2 rejection above and Haine further discloses wherein executing the application also causes the processing device (1020) to generate a listing of the graphic objects in the canvas, and sizes and locations of the graphic objects (as shown in Fig. 4C, the graphic objects 406A have size and locations assigned on the canvas in dashed lines).
Regarding claim 6, modified Haine teaches as is set forth in claim 5 rejection above and Haine further discloses wherein, for at least one of the graphic objects, the listing includes multiple locations for the graphic object (as shown in Fig. 5B, there are multiple locations for the graphic object as identified by Tool 503A-503D).
Regarding claim 7, modified Haine teaches as is set forth in claim 5 rejection above and Haine further discloses wherein the real-time graphics module is further configured to receive an eye angle of the user's eye ([0031], “Data derived from such eye movements may include timing and sequences of saccadic movements, eye direction (e.g., eye angle”); determine, based on the eye angle and on the sizes and locations of the graphic objects in the listing (as shown in Fig. 4C, size and location of graphic objects in the canvas affect what is visible in solid lines), which graphic objects fall within a display window of the display (as shown in Fig. 4C, based on angle of view only certain objects are displayed with bold lines); and transfer the images for those graphic objects from the canvas to the display ([0050], “the image projector effectively updates the SoE 304 relative to eye movements of a user by loading a corresponding portion of the virtual image and updating what is projected onto the eye.”).
Regarding claim 8, modified Haine teaches as is set forth in claim 1 rejection above and Haine further discloses further comprising:
a graphics engine that renders graphics objects into pixels ([0050], “the image projector that renders the recalled objects onto the retina”, examiner interpret the image projector to include a graphics engine as it renders the objects).
Regarding claim 9, modified Haine teaches as is set forth in claim 8 rejection above and Haine further discloses wherein the graphics engine renders text into pixels (as shown in Figs. 7A-7B, text is rendered into pixels for display as shown by 740).
Regarding claim 10, modified Haine teaches as is set forth in claim 1 rejection above and Haine further discloses wherein the processing device populates the canvas by transferring images of pre-rendered graphic objects to the canvas ([0050], “Objects 406A and partial objects 406B in FIG. 4C that do not fall within visible section 310 are not displayed on the retina and, thus remain invisible to the eye until being recalled from computer memory”, examiner interprets the objects in computer memory to be pre-rendered objects).
Regarding claim 12, modified Haine teaches as is set forth in claim 1 rejection above and Haine further discloses wherein the graphic objects are two-dimensional flat objects at a fixed distance and of a fixed size (as shown in Fig. 4C, the graphic objects are flat and of a fixed distance and size, [0042], “a canvas that has a fixed width and height that define the user's field of view, the entire world around the user's head/eyes is the virtual canvas”).
Regarding claim 16, modified Haine teaches as is set forth in claim 1 rejection above and Haine further discloses wherein, the real-time graphics module refreshes the display at a frame rate of the display by transferring from the canvas the images for all graphic objects that fall within a display window of the display ([0037], “In embodiments, mathematical transformations may be utilized to facilitate transitions between reference frames, coordinates, and parameters to enable smooth transitions at suitable rates when a user visually navigates and interacts with virtual objects in the coordinate space of a virtual scene”).
Regarding claim 20, modified Haine teaches as is set forth in claim 1 rejection above and Haine further discloses further comprising:
a wireless link to outside the eye-mounted device ([0027], “transceivers 115 for communication with internal and/or external devices”).
Regarding claim 21, modified Haine teaches as is set forth in claim 20 rejection above and Haine further discloses wherein operation of the real-time graphics module does not use the wireless link ([0075], “The manager 1000 may be implemented in an EMD”, examiner interprets this to mean the operations are done on device).
Regarding claim 22, modified Haine teaches as is set forth in claim 20 rejection above and Haine further discloses wherein the wireless link is used to upload the application onto the eye-mounted device ([0075], “any other device that controls various eye-tracking features that enable a user to activate, select and dismiss virtual objects within a virtual scene.”).
Regarding claim 24, modified Haine teaches as is set forth in claim 20 rejection above but does not specifically disclose further comprising: a contact lens that contains the display, the memory, the processing device and the real-time graphics module.
However Shtukater, in the same field of endeavor because both teach an eye-mounted device, teaches further comprising: a contact lens ([0178], “an active contact lens”) that contains the display (Fig. 9, element 903), memory ([0185], “Communication module 906 may include onboard memory”), the memory ([0185], “Communication module 906 may include onboard memory”), the processing device ([0187], “processor module 910”) and the real-time graphics module ([0191], “the processor module 910 may output visual information to the embedded display device 903”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater with the further comprising: a contact lens that contains the display, the memory, the processing device and the real-time graphics module as taught by Shtukater, for the purpose of creating a better augmented reality experience ([0191]).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Haine (US 2021/0208674) in view of Shtukater (US 2019/0179165), further in view of Deering (US 2015/0049004).
Regarding claim 4, modified Haine teaches as is set forth in claim 2 rejection above but does not specifically disclose wherein the real-time graphics module refreshes the display at a frame rate of at least 100 frames per second.
However Deering, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the real-time graphics module refreshes the display at a frame rate of at least 100 frames per second ([0372], “The advantage of 120 Hz display is that the existing 60 Hz frame transport video formats can be used to move the frame from the playback device (HDTV sources) to the display.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater with the wherein the real-time graphics module refreshes the display at a frame rate of at least 100 frames per second as taught by Deering, for the purpose of avoiding negative artifacts ([0372]).
Claims 11 and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Haine (US 2021/0208674) in view of Shtukater (US 2019/0179165), further in view of Haine2 (WO 2021080926 A1).
Regarding claim 11, modified Haine teaches as is set forth in claim 1 rejection above but does not specifically disclose wherein, for at least one graphic object, the processing device populates the canvas with images of at least two different versions of the graphic object.
However Haine2, in the same field of endeavor because both teach an eye-mounted device, teaches wherein, for at least one graphic object (Fig. 16E, element 1652), the processing device populates the canvas with images of at least two different versions of the graphic object (as shown in Fig. 16E, different versions of 1652 are populating the canvas (elements in dashed lines)).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater with the wherein, for at least one graphic object, the processing device populates the canvas with images of at least two different versions of the graphic object as taught by Haine2, for the purpose of making it easier to detect eye-motion ([00161]).
Regarding claim 13, modified Haine teaches as is set forth in claim 1 rejection above but does not specifically disclose wherein the processing device populates the canvas with images of all graphic objects used by the application prior to the transfer of images by the real-time graphics module.
However Haine2, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the processing device populates the canvas (examiner interprets the canvas to be everything outside of the visible section 310 in Fig. 18A-18B) with images of all graphic objects used by the application prior to the transfer of images by the real-time graphics module (as shown in Figs. 18A-18B, the graphic objects have all been populated on the canvas outside of 310).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater with the wherein the processing device populates the canvas with images of all graphic objects used by the application prior to the transfer of images by the real-time graphics module as taught by Haine2, for the purpose of providing an experience of looking around in a virtual scene ([00166]).
Regarding claim 14, modified Haine teaches as is set forth in claim 1 rejection above but does not specifically disclose wherein the canvas is updated with images of new graphic objects during the operation of the real-time graphics module.
However Haine2, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the canvas (Fig. 21, element 2100) is updated with images of new graphic objects during the operation of the real-time graphics module ([00174], “If a different trigger element in table 2070 is subsequently invoked, such as “Children Songs,” sub-table 2180 may be removed, updated, or otherwise replaced with a different sub-table or content comprising sub-elements associated with that different trigger element", examiner interprets the replaced with different sub-table to correspond to new graphic objects in the canvas).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater with the wherein the canvas is updated with images of new graphic objects during the operation of the real-time graphics module as taught by Haine2, for the purpose of reducing eye or neck motion ([00175]).
Regarding claim 15, modified Haine teaches as is set forth in claim 14 rejection above but does not specifically disclose wherein the operation of the real-time graphics module has priority over the updating of the canvas.
However Haine2, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the operation of the real-time graphics module has priority over the updating of the canvas ([00175], “such that in response to detecting that the user looks at element 2182, sub-table 2180 may be automatically activated/displayed in virtual scene 2100”, examiner interprets this to mean that the real-time graphics has priority since elements are automatically activated/displayed).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater, further in view of Haine2 with the wherein the operation of the real-time graphics module has priority over the updating of the canvas as taught by Haine2, for the purpose of reducing eye or neck motion ([00175]).
Claims 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Haine (US 2021/0208674) in view of Shtukater (US 2019/0179165), further in view of Yoshida (US 2023/0032431).
Regarding claim 17, modified Haine teaches as is set forth in claim 1 rejection above but does not specifically disclose wherein the display comprises a display buffer that stores images transferred from the canvas to the display, and the display projects the images stored in the display buffer.
However Yoshida, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the display (Fig. 2, 100) comprises a display buffer ([0034], “a low-resolution region buffer 213”) that stores images transferred from the canvas to the display, and the display projects the images stored in the display buffer ([0035], “The low-resolution region buffer 213 is controlled by the column control circuit, and stores and loads the display data”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater with the wherein the display comprises a display buffer that stores images transferred from the canvas to the display, and the display projects the images stored in the display buffer as taught by Yoshida, for the purpose of improving image quality ([0058]).
Regarding claim 18, modified Haine teaches as is set forth in claim 17 rejection above but does not specifically disclose wherein the real-time graphics module shifts images in the display buffer based on changes in an eye angle of the user's eye.
However Yoshida, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the real-time graphics module shifts images in the display buffer based on changes in an eye angle of the user's eye ([0039], “Step S303 is processing of generating the fovea control signal 219 from the line-of-sight information received from the line-of-sight detector 202 and holding it … step S305 is a step in which the data generator 201 generates the display data corresponding to the region 103 of the display 100”, examiner interprets this to mean the buffered image shifts based on change in line of sight).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater further in view of Yoshida with the wherein the real-time graphics module shifts images in the display buffer based on changes in an eye angle of the user's eye as taught by Yoshida, for the purpose of improving image quality ([0058]).
Regarding claim 19, modified Haine teaches as is set forth in claim 17 rejection above but does not specifically disclose wherein the real-time graphics module does not refresh the display buffer if an eye angle of the user's eye does not change.
However Yoshida, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the real-time graphics module does not refresh the display buffer if an eye angle of the user's eye does not change ([0040], “Step S306 is processing of transferring the region flag signal 210 and the fovea control signal 219 to the video display 200 … step S307 is a step in which the data generator 201 generates the display data corresponding to the region 102 of the display 100”, examiner interprets this to mean that no line of sight change in S302 leads to no change in buffered image).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater further in view of Yoshida with the wherein the real-time graphics module does not refresh the display buffer if an eye angle of the user's eye does not change as taught by Yoshida, for the purpose of improving image quality ([0058]).
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Haine (US 2021/0208674) in view of Shtukater (US 2019/0179165), further in view of Hamrah (US 2021/0393957).
Regarding claim 23, modified Haine teaches as is set forth in claim 20 rejection above but does not specifically disclose wherein the wireless link is a Bluetooth low energy (BLE) link.
However Hamrah, in the same field of endeavor because both teach an eye-mounted device, teaches wherein the wireless link is a Bluetooth low energy (BLE) link ([0099], “the wireless transceiver 304 may also support short range communication protocols (including such protocols as Bluetooth® (classical Bluetooth), Bluetooth-Low-Energy® (BLE) protocol,”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the instant invention to have the eye-mounted device of Haine in view of Shtukater with the wherein the wireless link is a Bluetooth low energy (BLE) link as taught by Hamrah, for the purpose of communicating with near-by devices ([0099]).
Conclusion
The prior art made of record and not relied upon are considered pertinent to applicant’s disclosure. Karri (US 2022/0100462), Singh (US 2022/0015622), Mirjalili (US 2020/0166776), He (WO 2018109570 A1), teach an eye-mounted device comprising: a display that projects pixels onto a retina of a user's eye; a memory storing an application, and a processing device coupled to the memory to execute the application, wherein the display, memory, processing device and real-time graphics module are contained within the eye-mounted device.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW Y LEE whose telephone number is (571)272-3526. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pinping Sun can be reached at (571) 270 - 1284. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW Y LEE/Examiner, Art Unit 2872