DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 07/13/2026 have been fully considered but they are not persuasive. Note the arguments are in view of amendment. The amendment changes the scope of the claims, but the examiner believes the references of previous rejection still applies as discussed below.
Applicant argues on page 11 that:
“However, it is respectfully submitted that in Geric the image processing and thus the actual overlay is performed by image processor 57 and not the AR display 60 itself. Thus, it is respectfully submitted that the Geric reference does not disclose a composer device implemented in hardware that directly processes a live image pixel stream as claimed, but rather merely a common software solution.”
Examiner respectfully disagrees. In view of the amendments, a composer device now encompasses an image processor and computer monitor or computer monitor or an augmented-reality display (herein referred to as AR display), i.e. something that works on live data. There is nothing in claims that excludes the interpretation of image processor 57 along with computer monitor / AR display as the claimed composer device. The claimed steps are performed by the image processor along with computer monitor / AR display (as argued by applicant above that the image processor 57 performs this).
New claim 22 is rejected in view of new grounds of rejection
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-6, 8-10, 13-14, 16, 19-20 rejected under 35 U.S.C. 103 as being unpatentable over Geric [US 20230051869 A1] in view of Guha [US 20200396379 A1].
As per claim 1, Geric teaches an augmentation overlay device (Geric Fig 7), comprising:
an image source configured for providing live image data of a region of interest in a form of a live image pixel stream (Geric Fig 7, ¶0042 “Camera images from instrument 53 …” ¶0054-¶0055 CCD camera /visible pixel, ¶0051 “a real, live view of the scene around the wearer …”);
a device configured for providing overlay data (Geric ¶0052 “a thermal camera coupled with the harvesting instrument (e.g., carried along with the endoscope) so that the image data includes a temperature overlay based on thermograms captured by the thermal camera”);
a composer device (Geric Fig 7, items 57 and 60, ¶0012 “A display presents the video stream and overlay to a user. The display can be a computer monitor or an augmented-reality display, for example”. Hence, for this rejection, a composer device is interpreted as device comprising the control unit 57 and computer monitor / AR displays),
wherein the composer device is a hardware device, and a function of the hardware device is implemented in hardware (Geric ¶0042 “controller (e.g., image processor) 57,” ¶0045 “Augmented-reality display 60 may be comprised of a head-worn display, sometimes also referred to as “smart glass” or “smart glasses”. Further computer monitor is hardware. Hence the image processor with monitor /AR display is hardware),
wherein the composer device is configured for:
providing display data in the form of the live image pixel stream working directly on the live image pixel stream by overlaying the provided live image data with the provided overlay data (Augmented-reality display 60 is configured to render glyphs (e.g., text, symbols, colored overlays, etc.) …” ¶0059 “an AR display which includes a video overlay 91 (streaming a live endoscopic view from a visible-light camera).”. Since image processor works directly on image data from camera to provide the live views and overlays, the new limitation of “working directly on the live image pixel stream” is met as well)
determining a restricted area in the provided live image data (Geric ¶0052 “Restriction of temperature overlay 70 to just the significant locations can be obtained by 1) using pattern recognition … or 2) by only representing temperatures”),
wherein the provided overlay data comprises restricted area overlay data that is associated with the restricted area of the live image data and free overlay data that is associated with an area of the live image data outside of the restricted area, and restricting overlay via the restricted area overlay data (Geric Fig 9, Geric ¶0052 “the temperature overlay may be limited to relevant surfaces within the visible-light view such as the side branches, target vessel, and tunnel wall. Restriction of temperature overlay 70 to just the significant locations…” the relevant surface to where overlay is restricted corresponds to the claimed area of the live image data outside of the restricted area, and the other surfaces are restricted areas. The data corresponding to these regions are the free overlay data and the restricted area overlay data respectively); and
a display device configured for receiving and displaying the provided display data (Geric Fig 8 view field 61).
The only difference from the claim is that Geric does not teach receiving overlay (thermal data here), from a data server.
Guha, in a related field of capturing and overlaying data teaches receiving overlay (thermal data here), from a data server (Guha ¶0030 “the server-side sensor analytics system 134 may be configured to implement additional sensor data analytics functionality not implemented by the sensor analytics system 114. For example, the server-side sensor analytics system 134 may generate trend data that captures thermal conditions of the one or more objects of interest over time”).
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to modify the apparatus in Geric, by utilizing data from server for providing additional sensor analytics functionality like thermal trend data to be generated and displayed (Guha ¶0030-¶0031)
As per claim 5, Geric in view of Guha further teaches wherein the composer device comprises a mixing circuit that is configured for; providing the display data in the form of the live image pixel stream by overlaying the provided live image data with the provided overlay data (Geric ¶0045 “Augmented-reality display 60 may be comprised of a head-worn display, sometimes also referred to as “smart glass” or “smart glasses”, among other names.”, ¶0048 “Augmented-reality display 60 is configured to render glyphs .. and to render video in the viewfield”, Mixing unit required for superimposing); and not reading the restricted area overlay data (Geric ¶0052 “Restriction of temperature overlay 70 to just the significant locations can be obtained by 1) using pattern recognition…” only relevant surface overlay data is read for mixing /overlaying).
As per claim 6, Geric in view of Guha further teaches wherein the composer device comprises a mixing circuit that is configured for providing the display data in the form of the live image pixel stream by overlaying the provided live image data with the provided overlay data, and wherein the composer device is further configured for; discarding the restricted area overlay data (Geric ¶0052 “Preferably, the temperature overlay may be limited to relevant surfaces”, other overlay data is preferably discarded); or determining a transparency value/alpha value of the restricted area overlay data (this is an optional limitation and examiner does not choose this).
As per claim 8, Geric in view of Guha further teaches wherein the composer device is configured for only considering the restricted area overlay data, when an additional security procedure is successfully performed (Geric ¶0052 “Restriction of temperature overlay 70 to just the significant locations can be obtained by 1) using pattern recognition …or 2) by only representing temperatures over a predetermined threshold (e.g., 40° C.)” steps 1 and 2 are security procedures).
As per claim 9, Geric in view of Guha further teaches wherein the additional security procedure comprises a predetermined specific user interaction performed by a user (Geric ¶0063, Fig 19 step 10, manual identification of side branch).
As per claim 10, Geric in view of Guha further teaches wherein the additional security procedure comprises a complex enabling sequence (Geric Fig 19 step 10, ¶0057 “an automated analysis of images is used to identify optimal location … characterized by image analysis (e.g., pattern recognition and depth detection)” Automated pattern recognition is, or contains complex enabling sequences for determining the relevant surfaces for display).
As per claim 13, Geric in view of Guha further teaches wherein the image source comprises a medical imaging device (Geric ¶0003 “devices and methods for endoscopic…”).
As per claim 14, Geric in view of Guha further teaches wherein the overlay data comprises one or more of: sensor data, patient monitoring data, 3D visualisation data, overlay menu data and/or videoconference data (Geric ¶0048, ¶0052).
As per claims 16, 19-20 have limitations similar to claims 1, 5-6 and are rejected for same reasons as above.
Claims 2-4, 17-18 rejected under 35 U.S.C. 103 as being unpatentable over Geric in view of Guha as applied to claims 1, 16 above, and further in view of Raghoebardajal [US 20120262485 A1]
As per claims 2-4, 17-18, Geric in view of Guha does not expressly teach wherein each pixel of the live image pixel stream of the live image data comprises x-y- coordinates, wherein the restricted area of the live image pixel stream of the live image data is determined using predetermined restricted x-y-coordinates, and wherein the restricted area comprises a shape, wherein a dimension and a location of the shape in the region of interest is at least partly defined by the x-y-coordinates, or wherein the restricted area
Raghoebardajal, in the field of processing for augmented reality teaches wherein each pixel of the live image pixel stream of the live image data comprises x-y- coordinates (Raghoebardajal ¶0151 “forward mapping of [W] and [P] applied to the local coordinates x.sub.L, y.sub.L of pixels”),
wherein the restricted area of the live image pixel stream of the live image data is determined using predetermined restricted x-y-coordinates (Raghoebardajal ¶0157 “3020A'', 3020B'' and 3020C'' are restricted by a circular mask. This mask corresponds to the circular buttons 3020A-C seen in the augmented image layer. The masks may be used in one or both of two ways. Firstly, the mask may be used to restrict the mapping of pixels from the control region to the augmentation image layer by use of [W][P],”), and
wherein the restricted area comprises a shape (Raghoebardajal ¶0157 circular shaped masks),
wherein a dimension and a location of the shape in the region of interest is at least partly defined by the x-y-coordinates (Raghoebardajal ¶0129 “The control image has a predetermined size and each of a plurality of control interfaces has a predetermined region within the control image”), or
wherein the restricted area of the live image pixel stream of the live image data is determined using a mask layer (Raghoebardajal ¶0157 “Control regions 3020A'', 3020B'' and 3020C'' are restricted by a circular mask”).
Raghoebardajal provides schemes for interreacting seamlessly with virtual entities for on-screen interplay of live video and augmentation (Raghoebardajal ¶0008). Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to modify the apparatus / method in Geric in view of Guha, by integrating graphical augmentation image layer and plurality of control interfaces and processing as in Raghoebardajal, so as to allow a user of thermal management using AR system, to interact seamlessly with virtual entities.
Claim 11, 21 rejected under 35 U.S.C. 103 as being unpatentable over Geric in view of Guha as applied to claim 10, 16 above, and further in view of Watts [US 9802317 B1].
As per claims 11, 21, Geric in view of Guha does not expressly teach wherein the complex enabling sequence comprises a challenge response from the data server
Watts, in a field of collecting and providing service provider records, teaches wherein the complex enabling sequence comprises a challenge response from the data serverWatts ¶0068 “before providing the overlay, the service provider record system 106 may confirm that the requesting user and/or web page has rights to the content associated with elements of the overlay (e.g., by confirming that the requesting domain has rights and/or authenticating the identity of the user by confirming a password)”).
As per MPEP 2143.I.F “Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art” is a rationale to support obviousness. In the instant case the claim is directed to providing challenge response to a user, which is known to already exist in the field of collection and providing records as in Watts. Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to modify the apparatus in Geric in view of Guha, so as to achieve predictable results of preventing a user from interacting with and/or viewing elements with content that they do not have rights to view.
Claim 15 rejected under 35 U.S.C. 103 as being unpatentable over Geric in view of Guha as applied to claim 1 above, and further in view of Mattila [US 20100328344 A1].
As per claim 15, Geric in view of Guha does not expressly teach wherein the composer device is implemented at least partially in an ASIC or FPGA.
Mattila, in a related field of augmented reality user interface teaches wherein the composer device is implemented at least partially in an ASIC or FPGA (Mattis ¶0047).
As per MPEP 2143.I.A “Combining prior art elements according to known methods to yield predictable results” is a rationale to support obviousness. Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to modify the apparatus in Geric in view of Guha so as to achieve predictable result of implementing via software, hardware using prior art components.
Claim 22 rejected under 35 U.S.C. 103 as being unpatentable over Geric in view of Guha as applied to claim 1 above, and further in view of
As per claim 22, Geric in view of Guha does not expressly teach wherein the composer device comprises an external hardware switch, activatable or de-activatable by the user, activating or preventing overlay of the restricted area.
Summa, in a field of information handling systems, teaches wherein the composer device comprises an external hardware switch, activatable or de-activatable by the user, activating or preventing overlay (Summa ¶0034 “allowing user to input a custom activation area position (e.g., via I/O device/s 107)”, ¶0046 “depressing a particular designated keyboard hotkey (or designated combination of keyboard hotkeys) or by pressing or physically touching other hardware switch or sensor of I/O 107, such as a designated hardware toggle switch, button, touch pad or other type of sensor pad”, Fig 2, steps 214 or 216).
As per MPEP 2143.I.F “Known work in one field of endeavor may prompt variations of it for use in either the same field or a different one based on design incentives or other market forces if the variations are predictable to one of ordinary skill in the art” is a rationale to support obviousness. In the instant case, the claim is directed to providing physical switch for activation / deactivation, which is known to already exist in the field of information handling systems. Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to modify the apparatus in Geric in view of Guha, so as to achieve predictable results of selective activations and interactions.
Allowable Subject Matter
Claim 12 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. In combination with parent claims, none of the references of record applied individually or in combination teaches wherein the additional security procedure comprises a request for overlay that enables the composer device to consider the restricted area overlay data for a predetermined limited time and to disable overlay automatically if no additional security procedure is passed before the limited time times out.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OOMMEN JACOB whose telephone number is (571)270-5166. The examiner can normally be reached 8:00-4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANNE M KOZAK can be reached at 571-270-0552. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Oommen Jacob/ Primary Examiner, Art Unit 3797
Prosecution Insights