DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, and 10 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Osterhout et al. (U.S. Patent Application Publication 2019/0025588).
Regarding claim 1, Osterhout et al. discloses a head-mounted display device (Figs. 1 and 190), comprising: a first zoomable camera configured to capture a first image viewed by a left eye of a user in a target scene (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective); a second zoomable camera configured to capture a second image viewed by a right eye of the user in the target scene (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective); and a display configured to display a left-eye target image on a left-eye display unit of the display and display a right-eye target image on a right-eye display unit of the display, the left-eye target image being obtained after zoom-in processing is performed on a first image region of interest (ROI) of the user comprised in the first image, and the right-eye target image is obtained after zoom-in processing is performed on a second image ROI comprised in the second image (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target – in embodiments, the dual cameras may be arranged such that their respective fields of view overlap at a distance such that they both image a same portion of the target – the overlap allows the two images to be presented in the see-through display fields of view where the same portion of the imaged target is presented to the user’s two eyes for a combined 3D view of the target – in embodiments, the head-worn system may include an eye-imaging system that monitors where the user ius looking, the vergence of the eyes, the focal plane the user is viewing, etc. – in embodiments, the vergence of the eyes may be used to determine the focal plane that the user is focused on and the captured images may be shifted or otherwise processed to change the portion of overlap to correspond with the focal plane of interest; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective; paragraphs [0743] and [0744]).
Regarding claim 2, Osterhout et al. discloses all of the limitations as previously discussed with respect to claim 1 including that wherein a first camera zoom ratio used by the first zoomable camera to capture the first image is the same as or different from a second camera zoom ratio used by the second zoomable camera to capture the second image (Osterhout et al.: Figs. 1 and 190; paragraph [0743] – in embodiments, the zoom level of each of the dual zoom cameras 19002 may be synchronized such that they maintain the same level of zoom – this can be useful when capturing images of the target 19004 and presenting the two captured images/videos simultaneously in two separate see-through display systems – this generates a good 3D image/video of the target for the user – setting both of the dual cameras at the same zoom level can help with depth sensing as well; paragraph [0744] – in embodiments, the zoom level of each of the two cameras may be set at two separate levels – this can be useful in generating a combined image (e.g., combined in the user’s mind when seeing the two separate images in a see-through computer display) that is formed from the separate zoom levels – it can be used to capture from both cameras but only displaying the image captured from one camera at a time – this can provide an instantaneous change in zoom by switching which camera capture is presented); and the first camera zoom ratio and the second camera zoom ratio are separately controlled (Osterhout et al.: Figs. 1 and 190; paragraph [0739] – the zoom may be controlled by an interface mounted on the head-worn computer (e.g., a capacitive touch surface, swipe surface, rotary dial, button(s), an external device (e.g., finger mounted UI, wand), gesture(s), voice); paragraph [0743] – in embodiments, the zoom level of each of the dual zoom cameras 19002 may be synchronized such that they maintain the same level of zoom; paragraph [0744] – in embodiments, the zoom level of each of the two cameras may be set at two separate levels).
Regarding claim 10, Osterhout et al. discloses all of the limitations as previously discussed with respect to claim 1 including that wherein the head-mounted display device is a mixed reality (MR) helmet (Osterhout et al.: Figs. 1 and 190; paragraph [0205] – in embodiments, the optics may be packaged as contact lenses – in other embodiments, the HWC 102 may be in the form of a helmet with a see-through shield, sunglasses, safety glasses, goggles, a mask, fire helmet with see-through shield, police helmet with see-through shield, military helmet with see-through shield, utility form customized to certain work task (e.g. inventory control, logistics, repair, maintenance, etc.), and the like).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3 and 7-9 are rejected under 35 U.S.C. 103 as being unpatentable over Osterhout et al. (U.S. Patent Application Publication 2019/0025588) in view of Asaban et al. (U.S. Patent Application Publication 2024/0377640).
Regarding claim 3, Osterhout et al. discloses all of the limitations as previously discussed with respect to claim 1 including that wherein the head-mounted display device further comprises a processor (Osterhout et al.: Figs. 1 and 190; paragraph [0206] – the HWC 102 may also have a number of integrated computing facilities, such as an integrated processor; paragraph [0207] – in another example, the HWC 102 may have sensors that detect movement (e.g. a nod, head shake, and the like) including accelerometers, gyros and other inertial measurements, where the integrated processor may interpret the movement and generate a control command in response). However, Osterhout et al. fails to disclose that the processor is configured to separately perform image processing on the first image and the second image to obtain the left-eye target image and the right-eye target image; and the image processing comprises the zoom-in processing performed on the first image ROI and the second image ROI.
Referring to the Asaban et al. reference, Asaban et al. discloses a head-mounted display device, comprising: a processor that is configured to separately perform image processing on the first image and the second image to obtain the left-eye target image and the right-eye target image (Figs. 2 and 3; paragraph [0014] – in some embodiments, the at least one processor is further configured to determine focus values corresponding to the determined distances and, for each determined distance, apply the corresponding focus values to the left and right video cameras; paragraph [0015] – in some embodiments, the processor is further configured to determine a magnification value and to magnify the displayed images on the see-through display by the magnification value; paragraph [0020] – in some embodiments, the at least one processor is configured to determine horizontal shift values corresponding to the determined distance from the left video camera and from the right video camera to the ROI, and horizontally shift the display of each image of the images captured by the left and right video cameras on the see-through display by the corresponding horizontal shift value; paragraph [0035] – in some embodiments, the see-through display includes left and right near-eye displays – the processor may be configured to generate the stereoscopic image by presenting respective left and right magnified images of the ROI on the left and right near-eye displays, while applying a horizontal shift to the left and right magnified images based on a distance from the head-mounted unit to the ROI; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI); and the image processing comprises the zoom-in processing performed on the first image ROI and the second image ROI (Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have had the processor separately perform image processing on the first image and the second image to obtain the left-eye target image and the right-eye target image as disclosed by Asaban et al. in the device disclosed by Osterhout et al. in order to provide an improved versatility and ease in use in adjusting the display to accommodate, for example, the user’s pupil spacing, region of interest, and/or desired magnification.
Regarding claim 7, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claims 1 and 3 including that wherein the processor is configured to: perform a binocular disparity adjustment on the first image and the second image to obtain a left- eye display view and a right-eye display view, the binocular disparity adjustment being based on a distance between a left-eye pupil and a right-eye pupil of the user and on positions of the first zoomable camera and the second zoomable camera on the head-mounted display device (Osterhout et al.: paragraph [0609] – in addition, the optical modules can be mounted in the frame of the head-worn display such that they are slightly pointed toward one another (also known as toe-in) to provide a convergence distance – thus, the convergence distance is established by the structural setup of the optics in the head-worn display and vergence distance can be adjusted by lateral digital shifting of similar portions of the left and right images that are displayed to create disparity for a portion of an image – the convergence distance then establishes the baseline vergence distance perceived by the user for stereo images that are rendered without disparity – to provide an improved stereo viewing experience, the convergence distance associated with the structural setup of the optics must be taken into account when rendering disparity associated with displayed objects in stereo images – this is particularly important in a head-worn display system wherein the focus distance and vergence distance are matched for augmented reality objects in stereo images; Asaban et al.: Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view); perform zoom-in processing on the first image ROI in the left-eye display view to obtain the left- eye target image (Osterhout et al.: paragraph [0609] – in addition, the optical modules can be mounted in the frame of the head-worn display such that they are slightly pointed toward one another (also known as toe-in) to provide a convergence distance – thus, the convergence distance is established by the structural setup of the optics in the head-worn display and vergence distance can be adjusted by lateral digital shifting of similar portions of the left and right images that are displayed to create disparity for a portion of an image – the convergence distance then establishes the baseline vergence distance perceived by the user for stereo images that are rendered without disparity – to provide an improved stereo viewing experience, the convergence distance associated with the structural setup of the optics must be taken into account when rendering disparity associated with displayed objects in stereo images – this is particularly important in a head-worn display system wherein the focus distance and vergence distance are matched for augmented reality objects in stereo images; Asaban et al.: Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view); and perform zoom-in processing on the second image ROI in the right-eye display view to obtain the right-eye target image (Osterhout et al.: paragraph [0609] – in addition, the optical modules can be mounted in the frame of the head-worn display such that they are slightly pointed toward one another (also known as toe-in) to provide a convergence distance – thus, the convergence distance is established by the structural setup of the optics in the head-worn display and vergence distance can be adjusted by lateral digital shifting of similar portions of the left and right images that are displayed to create disparity for a portion of an image – the convergence distance then establishes the baseline vergence distance perceived by the user for stereo images that are rendered without disparity – to provide an improved stereo viewing experience, the convergence distance associated with the structural setup of the optics must be taken into account when rendering disparity associated with displayed objects in stereo images – this is particularly important in a head-worn display system wherein the focus distance and vergence distance are matched for augmented reality objects in stereo images; Asaban et al.: Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view).
Regarding claim 8, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claims 1, 3, and 7 including that wherein the image processing further comprises image enhancement processing for the left-eye display view and the right-eye display view; and the image enhancement processing comprises at least one of the following: image sharpening, image dehazing, image deraining, image deblurring, image demosaicing, image contrast enhancement, image color enhancement, image detail enhancement, or image brightness enhancement (Osterhout et al.: paragraph [0549] – in yet a further embodiment, adjustments to attributes of the overall image can be made based on the local attributes of the portion of the displayed image or the portion of the see-through view that the user’s eye is looking at – the color adjusted attributes of the displayed image can include: color, color balance, contrast, sharpness, spatial frequency and resolution – where the eye camera is used to capture images of the user’s eye, which are then analyzed to determine the portion of the displayed image or the portion of the see-through view that the user’s eye is looking at – the portion of the displayed image or the portion of the see-through view that the user’s eye is looking at is then analyzed to determine the relative intensity of the attribute – adjustments are then made to the overall displayed image in correspondence to the local intensity of the attribute in the area that the user’s eye is looking at to improve viewability).
Regarding claim 9, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claims 1, 3, and 7 including that wherein the head-mounted display device further comprises an inertial measurement unit (IMU) configured to output IMU measurement data; and the processor is further configured to, when a head of the user is deflected, separately perform image stabilization processing on the left-eye display view and the right-eye display view, based on the IMU measurement data (Osterhout et al.: paragraph [0206] – the HWC 102 may also have a number of positional awareness sensors, such as GPS, electronic compass, altimeter, tilt sensor, IMU, and the like; paragraph [0493] – in embodiments, movement direction and speed of the head-mounted display is detected by the IMU sensor immediately prior to the display of each full color frame image – if the movement speed is above a predetermined threshold, the sequentially displayed color subframes associated with each full color frame are digitally shifted relative to one another so that they are displayed in an aligned position within the display field of view – the magnitude of the shift corresponds to the speed of the detected movement and the direction of the shift is counter to the detected direction of movement; paragraph [0710] – eye movements can be detected for example with an eye camera that captures images of the user’s eye while viewing the displayed image or by detecting changes in electric fields associated with the eye – angular movements of the user’s head can be detected relative to the world, relative to the user’s body through a motion sensor (e.g. IMU), etc. – fixing the displayed image in relation to the environment is good for viewing a wide angle image when the user is sitting or standing still – fixing the displayed image in relation to the user’s body is good for viewing a wide angle image when the user is walking, running, or riding in a vehicle).
Claims 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over Osterhout et al. (U.S. Patent Application Publication 2019/0025588) in view of Mao et al. (U.S. Patent Application Publication 2021/0365707).
Regarding claim 4, Osterhout et al. discloses all of the limitations as previously discussed with respect to claims 1 and 2, but fails to disclose that wherein the processor is further configured to: obtain the first camera zoom ratio and the second camera zoom ratio and determine the first image ROI and the second image ROI.
Referring to the Mao et al. reference, Mao et al. discloses a head-mounted display device, comprising: a processor that is configured to obtain the first camera zoom ratio and the second camera zoom ratio and determine the first image ROI and the second image ROI (Figs. 8B, 8C, 9A, 9B, 16, and 18; paragraph [0026] – in some aspects, the apparatus comprises a camera (e.g., an IP camera), a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wearable device, an extended reality device (e.g., a virtual (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device – in some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images – in some aspects, the apparatus further includes a display for displaying one or more images, notifications, and/or other displayable data; paragraph [0073] – in various scenarios (e.g., mobile imaging, video analytics, among other use cases), it can be desirable to maintain a size of a region of interest and/or object of interest (or target object) from frame-to-frame in sequence of frames (e.g., a video), even as the region of interest and/or object moves relative to one or more cameras capturing the sequence of frames; paragraph [0074] – image capture devices have increasing ranges of effective zoom – when a user is attempting to record a video of an object that is moving (e.g., a person playing soccer) and has already tuned the camera zoom so that the object has a desired size in the frame, the size ratio of the object (the size of the object relative to the frame, referred to as an object size-to-frame ratio) will dynamically change as the object moves; paragraph [0251] – Fig. 16 is a diagram illustrating a zooming process in a camera pipeline – a zoom region of interest (ROI) 1604 (also referred to as a cropping rectangle or a zoom rectangle) is shown in a frame 1602 that has a 1.0x zooming ratio – for instance, as described above, the ROI determination engine 804 of Fig. 8B can determine an initial region of interest based on user input and/or automatically; paragraph [0306] – the computing device can include any suitable device, such a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device of an autonomous vehicle, a robotic device, a television, and/or any other computing device with the resource capabilities to perform the processes described herein, including the process 820, the process 930, the process 935, the process 1800, and/or other process described herein – in some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component(s) that are configured to carry out the steps of processes described herein – in some cases, the computing device may include a display).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have had the processor obtain the first camera zoom ratio and the second camera zoom ratio and determine the first image ROI and the second image ROI as disclosed by Mao et al. in the device disclosed by Osterhout et al. in order to maintain fixed sizes for target objects in frames.
Regarding claim 5, Osterhout et al. in view of Mao et al. discloses all of the limitations as previously discussed with respect to claims 1, 2, and 4 including that wherein the processor is configured to: determine, based on the first camera zoom ratio and the second camera zoom ratio, one or more central picture regions corresponding to the first camera zoom ratio and the second camera zoom ratio from one or more shooting ranges of the first zoomable camera and the second zoomable camera, wherein the one or more central picture regions are used as the first image ROI and the second image ROI (Mao et al.: Figs. 8B, 8C, 9A, 9B, 16, and 18; paragraph [0011] – in some aspects, the point of the object region is a center point of the object region – in some cases, the object region is a bounding box (or other bounding region) – the center point can be a center point of the bounding box (or other region), a center point of the object (e.g., the object’s center of mass or center point); paragraph [0026] – in some aspects, the apparatus comprises a camera (e.g., an IP camera), a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wearable device, an extended reality device (e.g., a virtual (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device – in some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images – in some aspects, the apparatus further includes a display for displaying one or more images, notifications, and/or other displayable data; paragraph [0073] – in various scenarios (e.g., mobile imaging, video analytics, among other use cases), it can be desirable to maintain a size of a region of interest and/or object of interest (or target object) from frame-to-frame in sequence of frames (e.g., a video), even as the region of interest and/or object moves relative to one or more cameras capturing the sequence of frames; paragraph [0074] – image capture devices have increasing ranges of effective zoom – when a user is attempting to record a video of an object that is moving (e.g., a person playing soccer) and has already tuned the camera zoom so that the object has a desired size in the frame, the size ratio of the object (the size of the object relative to the frame, referred to as an object size-to-frame ratio) will dynamically change as the object moves; paragraph [0230] – by setting the object center point (or other point) and diagonal length (or other distance) of the bounding box, the process 1300 can initialize the target object information including the object center point coordinates, the object bounding box diagonal length, and the current zooming ratio for the object; paragraph [0251] – Fig. 16 is a diagram illustrating a zooming process in a camera pipeline – a zoom region of interest (ROI) 1604 (also referred to as a cropping rectangle or a zoom rectangle) is shown in a frame 1602 that has a 1.0x zooming ratio – for instance, as described above, the ROI determination engine 804 of Fig. 8B can determine an initial region of interest based on user input and/or automatically; paragraph [0306] – the computing device can include any suitable device, such a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device of an autonomous vehicle, a robotic device, a television, and/or any other computing device with the resource capabilities to perform the processes described herein, including the process 820, the process 930, the process 935, the process 1800, and/or other process described herein – in some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component(s) that are configured to carry out the steps of processes described herein – in some cases, the computing device may include a display).
Regarding claim 6, Osterhout et al. in view of Mao et al. discloses all of the limitations as previously discussed with respect to claims 1, 2, 4, and 5 including that wherein the processor is configured to: determine, according to an eye tracking algorithm, the first image ROI and the second image ROI from the one or more shooting ranges of the first zoomable camera and the second zoomable camera (Osterhout et al.: paragraph [0375] – in embodiments, the process involves collecting eye and/or sight heading information from a plurality of head-worn computers that come into proximity with an object in an environment – for example, a number of people may be walking through an area and each of the people may be wearing a head worn computer with the ability to track the position of the wearer’s eye(s) as well as the possibly the wearer’s sight and movement headings; paragraph [0472] – Figs. 97 and 98 show illustrations of optics modules similar to those shown in Fig. 94 but with the addition of an eye imaging camera 979 for capturing images of the user’s eye 9310 during use – the eye imaging camera 979 can be used to capture still images or video – where video images can be used to track movements of the user’s eye when looking at displayed images or when looking at a see-through view of the environment; paragraph [0664] – the head-worn display may further include a processor that is adapted to track an eye position of the user, the processor further adapted to alter a position of content as presented in the secondary display – the altered position may substantially maintain an alignment of the main image display and the secondary image display from the user’s perspective as the user’s eye moves).
Claims 11, 12, and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Osterhout et al. (U.S. Patent Application Publication 2019/0025588) in view of Asaban et al. (U.S. Patent Application Publication 2024/0377640).
Regarding claim 11, Osterhout et al. discloses an image processing method for a head-mounted display device (Figs. 1 and 190), the head-mounted display device comprising a first zoomable camera, a second zoomable camera, and a display (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective), the image processing method comprising: determining a first image region of interest (ROI) and a second image ROI in a target scene (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target – in embodiments, the dual cameras may be arranged such that their respective fields of view overlap at a distance such that they both image a same portion of the target – the overlap allows the two images to be presented in the see-through display fields of view where the same portion of the imaged target is presented to the user’s two eyes for a combined 3D view of the target – in embodiments, the head-worn system may include an eye-imaging system that monitors where the user ius looking, the vergence of the eyes, the focal plane the user is viewing, etc. – in embodiments, the vergence of the eyes may be used to determine the focal plane that the user is focused on and the captured images may be shifted or otherwise processed to change the portion of overlap to correspond with the focal plane of interest; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective; paragraphs [0743] and [0744]); and capturing, via the first zoomable camera, a first image viewed by a left eye of a user in the target scene and capturing, via the second zoomable camera, a second image viewed by a right eye of the user in the target scene (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective). However, Osterhout et al. fails to disclose separately performing image processing on the first image to obtain a left-eye target image and performing image processing on the second image to obtain a right-eye target image, the image processing comprising zoom-in processing performed on the first image ROI and on the second image ROI; and displaying the left-eye target image on a left-eye display unit of the display and displaying the right-eye target image on a right-eye display unit of the display.
Referring to the Asaban et al. reference, Asaban et al. discloses an image processing method for a head-mounted display device, comprising: separately performing image processing on the first image to obtain a left-eye target image and performing image processing on the second image to obtain a right-eye target image, the image processing comprising zoom-in processing performed on the first image ROI and on the second image ROI (Figs. 2 and 3; paragraph [0014] – in some embodiments, the at least one processor is further configured to determine focus values corresponding to the determined distances and, for each determined distance, apply the corresponding focus values to the left and right video cameras; paragraph [0015] – in some embodiments, the processor is further configured to determine a magnification value and to magnify the displayed images on the see-through display by the magnification value; paragraph [0020] – in some embodiments, the at least one processor is configured to determine horizontal shift values corresponding to the determined distance from the left video camera and from the right video camera to the ROI, and horizontally shift the display of each image of the images captured by the left and right video cameras on the see-through display by the corresponding horizontal shift value; paragraph [0035] – in some embodiments, the see-through display includes left and right near-eye displays – the processor may be configured to generate the stereoscopic image by presenting respective left and right magnified images of the ROI on the left and right near-eye displays, while applying a horizontal shift to the left and right magnified images based on a distance from the head-mounted unit to the ROI; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI); and displaying the left-eye target image on a left-eye display unit of the display and displaying the right-eye target image on a right-eye display unit of the display (Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have had the processor separately perform image processing on the first image and the second image to obtain the left-eye target image and the right-eye target image as disclosed by Asaban et al. in the method disclosed by Osterhout et al. in order to provide an improved versatility and ease in use in adjusting the display to accommodate, for example, the user’s pupil spacing, region of interest, and/or desired magnification.
Regarding claim 12, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claim 11 including that wherein a first camera zoom ratio used by the first zoomable camera to capture the first image is the same as or different from a second camera zoom ratio used by the second zoomable camera to capture the second image (Osterhout et al.: Figs. 1 and 190; paragraph [0743] – in embodiments, the zoom level of each of the dual zoom cameras 19002 may be synchronized such that they maintain the same level of zoom – this can be useful when capturing images of the target 19004 and presenting the two captured images/videos simultaneously in two separate see-through display systems – this generates a good 3D image/video of the target for the user – setting both of the dual cameras at the same zoom level can help with depth sensing as well; paragraph [0744] – in embodiments, the zoom level of each of the two cameras may be set at two separate levels – this can be useful in generating a combined image (e.g., combined in the user’s mind when seeing the two separate images in a see-through computer display) that is formed from the separate zoom levels – it can be used to capture from both cameras but only displaying the image captured from one camera at a time – this can provide an instantaneous change in zoom by switching which camera capture is presented); and the first camera zoom ratio and the second camera zoom ratio are separately controlled (Osterhout et al.: Figs. 1 and 190; paragraph [0739] – the zoom may be controlled by an interface mounted on the head-worn computer (e.g., a capacitive touch surface, swipe surface, rotary dial, button(s), an external device (e.g., finger mounted UI, wand), gesture(s), voice); paragraph [0743] – in embodiments, the zoom level of each of the dual zoom cameras 19002 may be synchronized such that they maintain the same level of zoom; paragraph [0744] – in embodiments, the zoom level of each of the two cameras may be set at two separate levels).
Regarding claim 14, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claim 11 including that wherein the determining the first image ROI and the second image ROI in the target scene comprises: determining, according to an eye tracking algorithm, the first image ROI and the second image ROI from one or more shooting ranges of the first zoomable camera and the second zoomable camera (Osterhout et al.: paragraph [0375] – in embodiments, the process involves collecting eye and/or sight heading information from a plurality of head-worn computers that come into proximity with an object in an environment – for example, a number of people may be walking through an area and each of the people may be wearing a head worn computer with the ability to track the position of the wearer’s eye(s) as well as the possibly the wearer’s sight and movement headings; paragraph [0472] – Figs. 97 and 98 show illustrations of optics modules similar to those shown in Fig. 94 but with the addition of an eye imaging camera 979 for capturing images of the user’s eye 9310 during use – the eye imaging camera 979 can be used to capture still images or video – where video images can be used to track movements of the user’s eye when looking at displayed images or when looking at a see-through view of the environment; paragraph [0664] – the head-worn display may further include a processor that is adapted to track an eye position of the user, the processor further adapted to alter a position of content as presented in the secondary display – the altered position may substantially maintain an alignment of the main image display and the secondary image display from the user’s perspective as the user’s eye moves).
Regarding claim 15, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claim 11 including that wherein the separately performing image processing on the first image and the second image to obtain the left-eye target image and the right-eye target image comprises: separately performing binocular disparity adjustment on the first image and the second image based on a distance between a left-eye pupil and a right-eye pupil of the user and based on positions of the first zoomable camera and the second zoomable camera on the head-mounted display device to obtain a left-eye display view and a right-eye display view (Osterhout et al.: paragraph [0609] – in addition, the optical modules can be mounted in the frame of the head-worn display such that they are slightly pointed toward one another (also known as toe-in) to provide a convergence distance – thus, the convergence distance is established by the structural setup of the optics in the head-worn display and vergence distance can be adjusted by lateral digital shifting of similar portions of the left and right images that are displayed to create disparity for a portion of an image – the convergence distance then establishes the baseline vergence distance perceived by the user for stereo images that are rendered without disparity – to provide an improved stereo viewing experience, the convergence distance associated with the structural setup of the optics must be taken into account when rendering disparity associated with displayed objects in stereo images – this is particularly important in a head-worn display system wherein the focus distance and vergence distance are matched for augmented reality objects in stereo images; Asaban et al.: Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view); performing zoom-in processing on the first image ROI in the left-eye display view to obtain the left-eye target image (Osterhout et al.: paragraph [0609] – in addition, the optical modules can be mounted in the frame of the head-worn display such that they are slightly pointed toward one another (also known as toe-in) to provide a convergence distance – thus, the convergence distance is established by the structural setup of the optics in the head-worn display and vergence distance can be adjusted by lateral digital shifting of similar portions of the left and right images that are displayed to create disparity for a portion of an image – the convergence distance then establishes the baseline vergence distance perceived by the user for stereo images that are rendered without disparity – to provide an improved stereo viewing experience, the convergence distance associated with the structural setup of the optics must be taken into account when rendering disparity associated with displayed objects in stereo images – this is particularly important in a head-worn display system wherein the focus distance and vergence distance are matched for augmented reality objects in stereo images; Asaban et al.: Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view); and performing zoom-in processing on the second image ROI in the right-eye display view to obtain the right-eye target image (Osterhout et al.: paragraph [0609] – in addition, the optical modules can be mounted in the frame of the head-worn display such that they are slightly pointed toward one another (also known as toe-in) to provide a convergence distance – thus, the convergence distance is established by the structural setup of the optics in the head-worn display and vergence distance can be adjusted by lateral digital shifting of similar portions of the left and right images that are displayed to create disparity for a portion of an image – the convergence distance then establishes the baseline vergence distance perceived by the user for stereo images that are rendered without disparity – to provide an improved stereo viewing experience, the convergence distance associated with the structural setup of the optics must be taken into account when rendering disparity associated with displayed objects in stereo images – this is particularly important in a head-worn display system wherein the focus distance and vergence distance are matched for augmented reality objects in stereo images; Asaban et al.: Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view).
Regarding claim 16, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claims 11 and 14 including that wherein the image processing further comprises image enhancement processing for the left-eye display view and the right-eye display view; and the image enhancement processing comprises at least one of the following: image sharpening, image dehazing, image deraining, image deblurring, image demosaicing, image contrast enhancement, image color enhancement, image detail enhancement, or image brightness enhancement (Osterhout et al.: paragraph [0549] – in yet a further embodiment, adjustments to attributes of the overall image can be made based on the local attributes of the portion of the displayed image or the portion of the see-through view that the user’s eye is looking at – the color adjusted attributes of the displayed image can include: color, color balance, contrast, sharpness, spatial frequency and resolution – where the eye camera is used to capture images of the user’s eye, which are then analyzed to determine the portion of the displayed image or the portion of the see-through view that the user’s eye is looking at – the portion of the displayed image or the portion of the see-through view that the user’s eye is looking at is then analyzed to determine the relative intensity of the attribute – adjustments are then made to the overall displayed image in correspondence to the local intensity of the attribute in the area that the user’s eye is looking at to improve viewability).
Regarding claim 17, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claims 11 and 14 including that wherein the head-mounted display device further comprises an inertial measurement unit (IMU), and the method further comprises: obtaining IMU measurement data output by the inertial measurement unit IMU; and when a head of the user is deflected, separately performing image stabilization processing on the left-eye display view and the right-eye display view based on the IMU measurement data (Osterhout et al.: paragraph [0206] – the HWC 102 may also have a number of positional awareness sensors, such as GPS, electronic compass, altimeter, tilt sensor, IMU, and the like; paragraph [0493] – in embodiments, movement direction and speed of the head-mounted display is detected by the IMU sensor immediately prior to the display of each full color frame image – if the movement speed is above a predetermined threshold, the sequentially displayed color subframes associated with each full color frame are digitally shifted relative to one another so that they are displayed in an aligned position within the display field of view – the magnitude of the shift corresponds to the speed of the detected movement and the direction of the shift is counter to the detected direction of movement; paragraph [0710] – eye movements can be detected for example with an eye camera that captures images of the user’s eye while viewing the displayed image or by detecting changes in electric fields associated with the eye – angular movements of the user’s head can be detected relative to the world, relative to the user’s body through a motion sensor (e.g. IMU), etc. – fixing the displayed image in relation to the environment is good for viewing a wide angle image when the user is sitting or standing still – fixing the displayed image in relation to the user’s body is good for viewing a wide angle image when the user is walking, running, or riding in a vehicle).
Regarding claim 18, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claim 11 including that wherein the head-mounted display device is a mixed reality (MR) helmet (Osterhout et al.: Figs. 1 and 190; paragraph [0205] – in embodiments, the optics may be packaged as contact lenses – in other embodiments, the HWC 102 may be in the form of a helmet with a see-through shield, sunglasses, safety glasses, goggles, a mask, fire helmet with see-through shield, police helmet with see-through shield, military helmet with see-through shield, utility form customized to certain work task (e.g. inventory control, logistics, repair, maintenance, etc.), and the like).
Regarding claim 19, Osterhout et al. discloses a non-transitory computer-readable media storing computer instructions that configure at least one processor, upon execution of the instructions, to perform the following steps: determining a first image region of interest (ROI) and a second image ROI in a target scene (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target – in embodiments, the dual cameras may be arranged such that their respective fields of view overlap at a distance such that they both image a same portion of the target – the overlap allows the two images to be presented in the see-through display fields of view where the same portion of the imaged target is presented to the user’s two eyes for a combined 3D view of the target – in embodiments, the head-worn system may include an eye-imaging system that monitors where the user ius looking, the vergence of the eyes, the focal plane the user is viewing, etc. – in embodiments, the vergence of the eyes may be used to determine the focal plane that the user is focused on and the captured images may be shifted or otherwise processed to change the portion of overlap to correspond with the focal plane of interest; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective; paragraphs [0743] and [0744]); capturing, via a first zoomable camera, a first image viewed by a left eye of a user in the target scene and capturing, via a second zoomable camera, a second image viewed by a right eye of the user in the target scene (Figs. 1 and 190; paragraph [0739] – a head-worn computer system that includes dual cameras with zoom capabilities that provide a stereo view of a target – the stereo view may be provided by a providing two separate captured views of the target, as captured by the dual cameras, as an overlay in a see-through computer display mounted in the head-worn computer – the dual cameras are separated and each one may be mounted proximate to one of the eyes of the user – the separation should be enough such that the two perspective captured views of the target generate a stereoscopic 3D view of the target; paragraph [0740] – Fig. 190 illustrates a head-worn computer 102 with two zoom cameras 19002a and 19002b; paragraph [0741] – there are many uses for a head-worn computer with stereo zoom cameras – surgeons and other medical professionals, for example, can use the system to magnify an area of the body for inspection or during a procedure – the captured images from the dual cameras can be presented in two see-through computer displays mounted in the head-worn computer such that the magnified area appears to overlay the see-through view of the body – the zoom can be easily altered during inspection or a procedure making the process more effective). However, Osterhout et al. fails to disclose separately performing image processing on the first image to obtain a left-eye target image and performing image processing on the second image to obtain a right-eye target image, the image processing comprising zoom-in processing performed on the first image ROI and on the second image ROI; and displaying the left-eye target image on a left-eye display unit of the display and displaying the right-eye target image on a right-eye display unit of the display.
Referring to the Asaban et al. reference, Asaban et al. discloses an image processing method for a head-mounted display device, comprising: separately performing image processing on the first image to obtain a left-eye target image and performing image processing on the second image to obtain a right-eye target image, the image processing comprising zoom-in processing performed on the first image ROI and on the second image ROI (Figs. 2 and 3; paragraph [0014] – in some embodiments, the at least one processor is further configured to determine focus values corresponding to the determined distances and, for each determined distance, apply the corresponding focus values to the left and right video cameras; paragraph [0015] – in some embodiments, the processor is further configured to determine a magnification value and to magnify the displayed images on the see-through display by the magnification value; paragraph [0020] – in some embodiments, the at least one processor is configured to determine horizontal shift values corresponding to the determined distance from the left video camera and from the right video camera to the ROI, and horizontally shift the display of each image of the images captured by the left and right video cameras on the see-through display by the corresponding horizontal shift value; paragraph [0035] – in some embodiments, the see-through display includes left and right near-eye displays – the processor may be configured to generate the stereoscopic image by presenting respective left and right magnified images of the ROI on the left and right near-eye displays, while applying a horizontal shift to the left and right magnified images based on a distance from the head-mounted unit to the ROI; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI); and displaying the left-eye target image on a left-eye display unit of the display and displaying the right-eye target image on a right-eye display unit of the display (Figs. 2 and 3; paragraphs [0014], [0015], [0020], and [0035]; paragraph [0063] – in some embodiments, the processor generates and presents a magnified stereoscopic image on the see-through display, so that the user is able to see a magnified 3D-like view of the ROI – the head-mounted unit (e.g., over-the-head unit or eyewear) may comprise left and right video cameras, which are mounted such that once the HMD device is worn by a user, the cameras will be located in a symmetrical manner with respect to the user’s (wearer’s) nose – the processor generates the stereoscopic image based on the images captured by both the left and right video cameras – for stereoscopic viewing, the display may comprise left and right near-eye displays, which represent respective left and right images (e.g., non-magnified or magnified images) of the ROI in front of the user’s left and right eyes, respectively – in several implementations, the processor applies a shift (e.g., horizontal shift) to the left and right magnified images based on the distance from the head-mounted unit to the ROI; paragraph [0070] – Fig. 3 is a flow chart that schematically illustrates an example method for generating magnified images for presentation on displays 30 – to generate the magnified images that are presented on displays 30, camera(s) 43 (at an image capture step 55) capture and output image data with respect to FOV 22 to processor 45 and/or processor 52 – at a data selection step 56, the processor 45, 52 selects and/or crops the part of the image data corresponding to ROI 24 – according to some aspects, the processor 45, 52 may select and/or crop a central portion of the image; paragraph [0072] – based on the image information received from cameras 43, the processor 45, 52 (at an image display step 57) generates and outputs a magnified image of the ROI 24 for presentation on displays 30 – the magnified images presented on the left and right displays 30 may be shifted (e.g., horizontally shifted) to give healthcare professional 26 a better stereoscopic view).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have had the processor separately perform image processing on the first image and the second image to obtain the left-eye target image and the right-eye target image as disclosed by Asaban et al. in the method disclosed by Osterhout et al. in order to provide an improved versatility and ease in use in adjusting the display to accommodate, for example, the user’s pupil spacing, region of interest, and/or desired magnification.
Claim 13 are rejected under 35 U.S.C. 103 as being unpatentable over Osterhout et al. in view of Asaban et al. as applied to claims 11 and 12 above, and further in view of Mao et al. (U.S. Patent Application Publication 2021/0365707).
Regarding claim 13, Osterhout et al. in view of Asaban et al. discloses all of the limitations as previously discussed with respect to claims 11 and 12, but fails to disclose wherein the determining the first image ROI and the second image ROI in the target scene comprises: obtaining the first camera zoom ratio and the second camera zoom ratio; and determining one or more central picture regions corresponding to the first camera zoom ratio and the second camera zoom ratio from one or more shooting ranges of the first zoomable camera and the second zoomable camera, wherein the one or more central picture regions are used as the first image ROI and the second image ROI.
Referring to the Mao et al. reference, Mao et al. discloses an image processing method for a head-mounted display device, comprising: wherein the determining the first image ROI and the second image ROI in the target scene comprises: obtaining the first camera zoom ratio and the second camera zoom ratio (Figs. 8B, 8C, 9A, 9B, 16, and 18; paragraph [0026] – in some aspects, the apparatus comprises a camera (e.g., an IP camera), a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wearable device, an extended reality device (e.g., a virtual (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device – in some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images – in some aspects, the apparatus further includes a display for displaying one or more images, notifications, and/or other displayable data; paragraph [0073] – in various scenarios (e.g., mobile imaging, video analytics, among other use cases), it can be desirable to maintain a size of a region of interest and/or object of interest (or target object) from frame-to-frame in sequence of frames (e.g., a video), even as the region of interest and/or object moves relative to one or more cameras capturing the sequence of frames; paragraph [0074] – image capture devices have increasing ranges of effective zoom – when a user is attempting to record a video of an object that is moving (e.g., a person playing soccer) and has already tuned the camera zoom so that the object has a desired size in the frame, the size ratio of the object (the size of the object relative to the frame, referred to as an object size-to-frame ratio) will dynamically change as the object moves; paragraph [0251] – Fig. 16 is a diagram illustrating a zooming process in a camera pipeline – a zoom region of interest (ROI) 1604 (also referred to as a cropping rectangle or a zoom rectangle) is shown in a frame 1602 that has a 1.0x zooming ratio – for instance, as described above, the ROI determination engine 804 of Fig. 8B can determine an initial region of interest based on user input and/or automatically; paragraph [0306] – the computing device can include any suitable device, such a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device of an autonomous vehicle, a robotic device, a television, and/or any other computing device with the resource capabilities to perform the processes described herein, including the process 820, the process 930, the process 935, the process 1800, and/or other process described herein – in some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component(s) that are configured to carry out the steps of processes described herein – in some cases, the computing device may include a display); and determining one or more central picture regions corresponding to the first camera zoom ratio and the second camera zoom ratio from one or more shooting ranges of the first zoomable camera and the second zoomable camera, wherein the one or more central picture regions are used as the first image ROI and the second image ROI (Figs. 8B, 8C, 9A, 9B, 16, and 18; paragraph [0011] – in some aspects, the point of the object region is a center point of the object region – in some cases, the object region is a bounding box (or other bounding region) – the center point can be a center point of the bounding box (or other region), a center point of the object (e.g., the object’s center of mass or center point); paragraph [0026] – in some aspects, the apparatus comprises a camera (e.g., an IP camera), a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wearable device, an extended reality device (e.g., a virtual (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device – in some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images – in some aspects, the apparatus further includes a display for displaying one or more images, notifications, and/or other displayable data; paragraph [0073] – in various scenarios (e.g., mobile imaging, video analytics, among other use cases), it can be desirable to maintain a size of a region of interest and/or object of interest (or target object) from frame-to-frame in sequence of frames (e.g., a video), even as the region of interest and/or object moves relative to one or more cameras capturing the sequence of frames; paragraph [0074] – image capture devices have increasing ranges of effective zoom – when a user is attempting to record a video of an object that is moving (e.g., a person playing soccer) and has already tuned the camera zoom so that the object has a desired size in the frame, the size ratio of the object (the size of the object relative to the frame, referred to as an object size-to-frame ratio) will dynamically change as the object moves; paragraph [0230] – by setting the object center point (or other point) and diagonal length (or other distance) of the bounding box, the process 1300 can initialize the target object information including the object center point coordinates, the object bounding box diagonal length, and the current zooming ratio for the object; paragraph [0251] – Fig. 16 is a diagram illustrating a zooming process in a camera pipeline – a zoom region of interest (ROI) 1604 (also referred to as a cropping rectangle or a zoom rectangle) is shown in a frame 1602 that has a 1.0x zooming ratio – for instance, as described above, the ROI determination engine 804 of Fig. 8B can determine an initial region of interest based on user input and/or automatically; paragraph [0306] – the computing device can include any suitable device, such a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, an autonomous vehicle or computing device of an autonomous vehicle, a robotic device, a television, and/or any other computing device with the resource capabilities to perform the processes described herein, including the process 820, the process 930, the process 935, the process 1800, and/or other process described herein – in some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and/or other component(s) that are configured to carry out the steps of processes described herein – in some cases, the computing device may include a display).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have had the processor obtain the first camera zoom ratio and the second camera zoom ratio and determine the first image ROI and the second image ROI as disclosed by Mao et al. in the method disclosed by Osterhout et al. in view of Asaban et al. in order to maintain fixed sizes for target objects in frames.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HEATHER R JONES whose telephone number is (571)272-7368. The examiner can normally be reached Mon. - Fri.: 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Vaughn can be reached at (571)272-3922. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HEATHER R JONES/Primary Examiner, Art Unit 2481
August 8, 2026
Prosecution Insights