DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (“IDS”) filed on March 18, 2024 was reviewed and the listed references were noted.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 7, 12, 16, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1).
Regarding claim 1, Kuo teaches a computer system comprising:
a processor system (Kuo, Para. [0083], the term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers); and
a storage system comprising instructions that are executable by the processor system to cause the computer system (Kuo, Para. [0082], one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus) to:
obtain an image of an environment in which an object of interest (OOI) is located (Kuo, Para. [0042], an instance segmentation system that processes an image to detect and segment individual objects (i.e., object of interest) in the image), wherein the image includes pixel content representative of the OOI (Kuo, Para. [0045], the image can be represented as a 2D array of pixels),
perform target detection on the image by generating a bounding element that surrounds the pixel content that is representative of the OOI (Kuo, Para. [0044], the system 100 processes the image 102 to generate detection data 104 which defines regions in the image 102 (e.g., by bounding boxes) which depict respective objects);
subsequent to performing the target detection, perform target segmentation on pixels surrounded by the bounding element (Kuo, Para. [0044], For each of the image regions defined by the detection data 104 (i.e., bounding boxes), the system 100 generates a respective object segmentation 106 that defines whether each pixel in the region is included in the object depicted in the region), wherein the target segmentation identifies pixels that represent the OOI and pixels that do not represent the OOI, and wherein the target segmentation identifies a silhouette of the OOI (Kuo, Para. [0048], the system 100 generates a shape descriptor 116 that characterizes the estimated shape of the object (i.e., identifies a silhouette). More specifically, the shape descriptor 116 specifies a respective score for each pixel in the image region enclosing the object that characterizes a likelihood that the pixel is included in the object (i.e., identifies pixels that represent the OOI and pixels that do not represent the OOI). Para. [0049], pixels included in the object have one value (e.g., the value 1), and the remaining pixels have a different value (e.g., the value 0)).
Although Kuo teaches performing object segmentation on an image (Kuo, Para. [0042]), Kuo does not explicitly teach “the image being generated by a camera” and to “obtain pose data for the camera, the pose data representing a pose of the camera when the camera generated the image”. However, in an analogous field of endeavor, Castillo teaches the image capture device may be a built-in camera of a client device or a digital camera communicatively coupled to the client device (Castillo, Col. 28, lines 15-19). Castillo teaches a user device executes a native application to initiate an image capture session for generating a 3D model of a physical structure. The native application can capture a first 2D image of the physical structure from a first pose (Castillo, Col. 45, lines 28-57). The 3D model reconstruction system can triangulate camera poses to reconstruct the 3D model (Castillo, Col. 18, lines 1-19).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo with the teachings of Castillo by including capturing images using a camera and obtaining pose data for the camera that represents the pose of the camera when the image was captured. One having ordinary skill in the art before the effective filing date would have been motivated to combine these references because doing so would allow for enhancing image capture of physical objects, as recognized by Castillo.
Although Kuo in view of Castillo teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach to “determine whether a reticle associated with the camera overlaps the pixels that represent the OOI within the image”. However, in an analogous field of endeavor, Uscinski teaches the wearable system can use a variety of techniques to calculate alignments between the reticle and the eye calibration target (i.e., OOI). As one example, the wearable system can determine the relative positions between the reticle and the eye calibration target. If the eye calibration target is within the reticle or a portion of eye calibration target overlaps with the reticle, the wearable system can determine that the reticle has aligned with the eye calibration target. The wearable system can also determine that the reticle and the target are aligned if the center of the reticle and the target coincide sufficiently. The wearable system may determine whether the reticle and target overlap or coincide by determining that the relative offset between them is smaller than a threshold (e.g., an angular threshold as described above) (Uscinski, Para. [0145]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo with the teachings of Uscinski by including determining that a reticle overlaps the pixels of the target (i.e., OOI) in the image. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for sufficiently aligning a reticle with an object of interest, as recognized by Uscinski. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 2, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, wherein the image is one of a thermal image, a low light image, or a red-green-blue (RGB) image (Kuo, Para. [0045], if the image 102 is a red-green-blue (RGB) image, then each pixel can be represented as a vector with three integer or floating point components, which respectively represent the intensity of the red, green, and blue color of the pixel).
Regarding claim 3, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, wherein the pose data is generated using an inertial measurement unit (IMU) (Uscinski, Para. [0155], the wearable system can track the head poses using sensors internal to an HMD such as, e.g., an IMU. Para. [0093], the wearable system can include three pairs of cameras).
The proposed combination as well as the motivation for combining the Kuo, Castillo and Uscinski references presented in the rejection of Claim 1, apply to Claim 3 and are incorporated herein by reference. Thus, the system recited in Claim 3 is met by Kuo in view of Castillo further in view of Uscinski.
Regarding claim 7, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, wherein the instructions are further executable to cause the computer system to:
determine one or more environmental conditions that are present within the environment at a time when the image was generated (Uscinski, Para. [0078], the wearable systems can use various sensors (e.g., accelerometers, gyroscopes, temperature sensors, movement sensors, depth sensors, GPS sensors, inward-facing imaging system, outward-facing imaging system, etc.) to determine the location and various other attributes of the environment of the user).
The proposed combination as well as the motivation for combining the Kuo, Castillo and Uscinski references presented in the rejection of Claim 1, apply to Claim 7 and are incorporated herein by reference. Thus, the system recited in Claim 7 is met by Kuo in view of Castillo further in view of Uscinski.
Claim 12 recites a method with steps corresponding to the elements of the system recited in Claims 1. Therefore, the recited steps of this claim are mapped to the proposed combination in the same manner as the corresponding elements in its corresponding system claim. Additionally, the rationale and motivation to combine the Kuo, Castillo and Uscinski references, presented in rejection of Claim 1, apply to this claim.
Regarding claim 16, Kuo in view of Castillo further in view of Uscinski teaches the method of claim 12, wherein the method further includes transmitting a notice to a computer system associated with the OOI informing the computer system that the OOI has been targeted (Uscinski, Para. [0085], the object recognizers may crawl through these collected points and recognize one or more objects using a map database. This information may then be conveyed to the user's individual wearable system, and the desired virtual scene may be accordingly displayed to the user).
The proposed combination as well as the motivation for combining the Kuo, Castillo, and Uscinski references presented in the rejection of Claim 1, apply to Claim 16 and are incorporated herein by reference. Thus, the method recited in Claim 16 is met by Kuo in view of Castillo further in view of Uscinski.
Regarding claim 18, Kuo teaches a computer system comprising:
a processor system (Kuo, Para. [0083], the term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers; and
a storage system comprising instructions that are executable by the processor system to cause the computer system (Kuo, Para. [0082], one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus) to:
obtain an image of an environment in which an object of interest (OOI) is located (Kuo, Para. [0042], an instance segmentation system that processes an image to detect and segment individual objects (i.e., object of interest) in the image), wherein the image includes pixel content representative of the OOI (Kuo, Para. [0045], the image can be represented as a 2D array of pixels),
perform target detection on the image by generating a bounding element that surrounds the pixel content that is representative of the OOI (Kuo, Para. [0044], the system 100 processes the image 102 to generate detection data 104 which defines regions in the image 102 (e.g., by bounding boxes) which depict respective objects);
subsequent to performing the target detection, perform target segmentation on pixels surrounded by the bounding element (Kuo, Para. [0044], For each of the image regions defined by the detection data 104 (i.e., bounding boxes), the system 100 generates a respective object segmentation 106 that defines whether each pixel in the region is included in the object depicted in the region), wherein the target segmentation identifies pixels that represent the OOI and pixels that do not represent the OOI, and wherein the target segmentation identifies a silhouette of the OOI (Kuo, Para. [0048], the system 100 generates a shape descriptor 116 that characterizes the estimated shape of the object (i.e., identifies a silhouette). More specifically, the shape descriptor 116 specifies a respective score for each pixel in the image region enclosing the object that characterizes a likelihood that the pixel is included in the object (i.e., identifies pixels that represent the OOI and pixels that do not represent the OOI). Para. [0049], pixels included in the object have one value (e.g., the value 1), and the remaining pixels have a different value (e.g., the value 0)).
Although Kuo teaches performing object segmentation on an image (Kuo, Para. [0042]), Kuo does not explicitly teach “the image being generated by a camera” and to “obtain pose data for the camera, the pose data representing a pose of the camera when the camera generated the image”. However, in an analogous field of endeavor, Castillo teaches the image capture device may be a built-in camera of a client device or a digital camera communicatively coupled to the client device (Castillo, Col. 28, lines 15-19). Castillo teaches a user device executes a native application to initiate an image capture session for generating a 3D model of a physical structure. The native application can capture a first 2D image of the physical structure from a first pose (Castillo, Col. 45, lines 28-57). The 3D model reconstruction system can triangulate camera poses to reconstruct the 3D model (Castillo, Col. 18, lines 1-19).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo with the teachings of Castillo by including capturing images using a camera and obtaining pose data for the camera that represents the pose of the camera when the image was captured. One having ordinary skill in the art before the effective filing date would have been motivated to combine these references because doing so would allow for enhancing image capture of physical objects, as recognized by Castillo.
Although Kuo in view of Castillo teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach to “determine that a center portion of a reticle, which is associated with the camera, overlaps the pixels that represent the OOI within the image”, “display the image and the reticle on a display of the computer system”, and “display an indication that the OOI has been targeted as a result of the center portion of the reticle overlapping the pixels that represent the OOI at the specified instant in time”. However, in an analogous field of endeavor, Uscinski teaches the wearable system can use a variety of techniques to calculate alignments between the reticle and the eye calibration target (i.e., OOI). As one example, the wearable system can determine the relative positions between the reticle and the eye calibration target. If the eye calibration target is within the reticle or a portion of eye calibration target overlaps with the reticle, the wearable system can determine that the reticle has aligned with the eye calibration target. The wearable system can also determine that the reticle and the target are aligned if the center of the reticle and the target coincide sufficiently. The wearable system may determine whether the reticle and target overlap or coincide by determining that the relative offset between them is smaller than a threshold (e.g., an angular threshold as described above) (Uscinski, Para. [0145]). Uscinski further teaches a user can perceive a reticle and a target via the display (i.e., display the image and reticle) (Uscinski, Para. [0136]). Uscinski further teaches the wearable system can determine whether the reticle has aligned with the robot for a threshold amount of time. If the alignment between the reticle and the robot has been aligned over the threshold period of time, the display 220 can present a visual focus indicator (such as, e.g., a graphic 1830) which indicating that the robot has been destroyed (e.g., scene 1800d) (Uscinski, Para. [0192]; Fig. 18).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo with the teachings of Uscinski by including determining that a reticle overlaps the pixels of the target (i.e., OOI) in the image, displaying the reticle overlapping the target on a display, and display that the reticle has overlapped the target at a specified instant in time. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for sufficiently aligning a reticle with an object of interest, as recognized by Uscinski. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Deng et al. (US 2020/0302173 A1).
Regarding claim 4, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach “wherein performing the target segmentation is performed substantially in real-time”. However, in an analogous field of endeavor, Deng teaches a real-time object segmentation system including an image object segmenting unit for generating a first segmentation region (Deng, Para. [0046]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo further in view of Uscinski with the teachings of Deng by including performing the target segmentation in real-time. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for presenting a segmentation region to a user in real-time, as recognized by Deng. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Price et al. (US 2024/0371002 A1, filed May 5, 2023).
Regarding claim 5, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach “wherein performing the target segmentation further includes granularly classifying each of the pixels that represent the OOI as a particular part of the OOI”. However, in an analogous field of endeavor, Price teaches the segmentation module segments sub-objects of the object of interest (i.e., a particular part of the OOI). In some embodiments, a sub-object classifier optionally classifies a pixel as belonging to a segment (otherwise referred to herein as a sub-object) of the object of interest using generic labels (i.e., granularly classifying each of the pixels as a particular part). If executed, the sub-object classifier uses the mask embeddings to classify pixels as belonging to sub-objects according to a generic label, instead of classifying sub-objects according to a ground truth label. For example, the sub-object classifier classifies pixels of a particular sub-object as likely belonging to “part 1” without identifying what “part 1” is. Because the sub-object classifier is class agnostic, the accuracy of sub-object classifier does not depend on learning ground truth labels of sub-objects. That is, the sub-object classifier can classify unique sub-objects of an object of interest without such sub-objects being part of the training data (Price, Para. [0026]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo further in view of Uscinski with the teachings of Price by including granularly classifying each pixel into a sub-object (i.e., particular part of the OOI). One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for automatically performing detailed object segmentation, as recognized by Price. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Xiao et al. (US 2023/0177682 A1).
Regarding claim 6, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach “wherein the target segmentation includes a foreground-background segmentation stage”. However, in an analogous field of endeavor, Xiao teaches an instance segmentation algorithm that detects an object bounding box first and assigns pixels as foreground or background within this bounding box (i.e., a foreground-background segmentation stage) (Xiao, Para. [0164]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo further in view of Uscinski with the teachings of Xiao by including segmenting pixels in the bounding box into a foreground and background. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for automatically identifying objects through image segmentation techniques, as recognized by Xiao. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 20, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 18, wherein a position of the reticle is adjusted based on one or more detected environmental conditions detected within the environment (Uscinski, Para. [0183], a large virtual snowball may be anchored at a fixed location in the user's environment and, in this example, may be intended to represent the base portion of a snowman. A virtual target (shown with dashed lines) may also be anchored at a fixed location in the user's environment, and may correspond to a target position within the user's environment with which the user is to align a reticle).
The proposed combination as well as the motivation for combining the Kuo, Castillo, and Uscinski references presented in the rejection of Claim 1, apply to Claim 20 and are incorporated herein by reference. Thus, the system recited in Claim 20 is met by Kuo in view of Castillo further in view of Uscinski.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Jnawali et al. (US 2024/0119706, filed September 6, 2023).
Regarding claim 8, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach “wherein the target detection includes use of a deep learning algorithm comprising a you-only-look-once (YOLO) algorithm”. However, in an analogous field of endeavor, Jnawali teaches a deep learning YOLO-based object detector configured to: (1) receive, as input, only one input frame, and (2) detect and localize, using a YOLO-based object detection algorithm, one or more small objects displayed within the input frame, and (3) provide, as output, an output frame comprising the one or more small objects and its one or more bounding boxes (BB) (Jnawali, Para. [0047]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the system of Kuo in view of Castillo further in view of Uscinski with the teachings of Jnawali by including performing the target detection using a deep learning YOLO-based object detector. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for generating an enhanced image with reconstructed objects, as recognized by Jnawali. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Dai et al. (US 2019/0025584 A1).
Regarding claim 9, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches determining sparse points that can be used in displaying and understanding the orientation and position of various objects in the user’s surroundings (Uscinski, Para. [0085]), they do not explicitly teach “wherein the target detection includes cross-referencing the image with one or more of: global positioning system (GPS) data or orientation data of all OOIs, including said OOI, in the environment”. However, in an analogous field of endeavor, Dai teaches a light data structure which includes digital data that describes the geographical location of static points of interest such as traffic signals and traffic signs which can be queried or cross-referenced using the GPS data for the vehicle to identify whether the vehicle is near a point of interest described by the digital data stored in the light data structure (Dai, Para. [0014]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo further in view of Uscinski with the teachings of Dai by including cross-referencing geographical locations of points of interest (i.e., OOIs) with GPS data identifying location near a point of interest. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for generating an augmented reality system using object detection, as recognized by Dai. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claims 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Keller et al. (US 2023/0360290 A1).
Regarding claim 10, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 1, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches when the eye calibration target is within the reticle or a portion of eye calibration target overlaps with the reticle, the wearable system can determine that the reticle has aligned with the eye calibration target (Uscinski, Para. [0145]), they do not explicitly teach “wherein determining whether the reticle associated with the camera overlaps the pixels that represent the OOI within the image includes determining whether a center-point of the reticle is within the bounding element”. However, in an analogous field of endeavor, Keller teaches the bounding box may also contain the reticle indicating the center of the FOV (Keller, Para. [0052]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo with the teachings of Uscinski by including the determination of the reticle aligning with the eye calibration target as taught by Uscinski includes determining that the reticle is within the bounding element as taught by Keller. One having ordinary skill in the art before the effective filing date would have been motivated to combine these references because doing so would allow for an optical alignment system for calibrating a virtual image reticle, as recognized by Keller. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Regarding claim 11, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 10, wherein determining whether the reticle associated with the camera overlaps the pixels that represent the OOI within the image includes determining whether the center-point of the reticle is within the bounding element and is further overlapping the pixels that represent the OOI (Uscinski, Para. [0145], If the eye calibration target is within the reticle or a portion of eye calibration target overlaps with the reticle, the wearable system can determine that the reticle has aligned with the eye calibration target).
The proposed combination as well as the motivation for combining the Kuo, Castillo, Uscinski and Keller references presented in the rejection of Claim 10, apply to Claim 11 and are incorporated herein by reference. Thus, the system recited in Claim 11 is met by Kuo in view of Castillo further in view of Uscinski and Keller.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Halligan et al. (US 2018/0330165 A1).
Regarding claim 13, Kuo in view of Castillo further in view of Uscinski teaches the method of claim 12, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach “wherein the target segmentation includes use of an unsupervised energy-minimization graph cut algorithm to segment the OOI”. However, in an analogous field of endeavor, Halligan teaches using the Boykov-Kolmogorov algorithm to determine the likelihoods that the segments of an image represent plants (i.e., to segment the OOI) (Halligan, Para. [0040]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Kuo in view of Castillo further in view of Uscinski with the teachings of Halligan by including segmenting the OOI by using an unsupervised energy-minimization graph cut algorithm (i.e., Boykov-Kolmogorov). One having ordinary skill in the art before the effective filing date would have been motivated to combine these references because doing so would allow for dynamically improving the quality of images, as recognized by Halligan. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Vallez Enano et al. (US 2024/0171845 A1, filed November 21, 2023).
Regarding claim 14, Kuo in view of Castillo further in view of Uscinski teaches the method of claim 12, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches generating bounding boxes defining object detections (Kuo, Para. [0044]), they do not explicitly teach “wherein a position of the bounding element is set so that the OOI is centered in the bounding element”. However, in an analogous field of endeavor, Vallez Enano teaches generating a candidate object bounding box that is centered on the central point of the target object (i.e., OOI is centered in the bounding element) (Vallez Enano, Para. [0138]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the method of Kuo in view of Castillo further in view of Uscinski with the teachings of Vallez Enano by including positioning the bounding box so that the OOI is centered in the bounding element. One having ordinary skill in the art before the effective filing date would have been motivated to combine these references because doing so would allow for tracking an object in a camera field of view more accurately, as recognized by Vallez Enano. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Kalyanasundram et al. (US 2021/0035334 A1).
Regarding claim 15, Kuo in view of Castillo further in view of Uscinski teaches the method of claim 12, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches segmenting an object of interest in an image (Kuo, Para. [0048]), they do not explicitly teach “wherein the image has a thermal spectrum and is manifested as having foreground pixels and background pixels”. However, in an analogous field of endeavor, Kalyanasundram teaches converting a thermal image to a heatmap image (i.e., the image has a thermal spectrum) and generating a foreground image by separating a background represented in the heatmap image from a foreground represented in the heatmap image. As used herein, a “background” represents portions (i.e., pixels) of the heatmap image that do not include a heat source. For example, if the thermal image is of a room in a shared office space, the background in the heatmap image may include desks, cubicles, chairs, the floor, etc. As used herein, a “foreground” represents portions of the heatmap image that include a heat source (e.g., a person and/or any other object that generates heat) (Kalyanasundram, Para. [0037]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Kuo in view of Castillo further in view of Uscinski with the teachings of Kalyanasundram by including the image being a thermal image turned into a heat map for segmenting pixels into a foreground and background. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for segmenting objects in an image based on identifying a heat source, as recognized by Kalyanasundram. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Mahbub et al. (US 2022/0076059 A1).
Regarding claim 17, Kuo in view of Castillo further in view of Uscinski teaches the method of claim 12, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches generating bounding boxes defining object detections (Kuo, Para. [0044]), they do not explicitly teach “wherein the target detection further includes down-sampling at least some pixels surrounded by the bounding element”. However, in an analogous field of endeavor, Mahbub teaches a down-sampler (DS) operation may receive the bounding boxes and may provide a down-sampled image output (Mahbub, Para. [0061]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date to modify the method of Kuo in view of Castillo further in view of Uscinski with the teachings of Mahbub by including down-sampling the bounding boxes (i.e., some pixels surrounded by the bounding element). One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for analyzing regions of an image, as recognized by Mahbub. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Kuo et al. (US 2021/0374453 A1) in view of Castillo et al. (US 12,445,717 B2, filed January 29, 2021) further in view of Uscinski et al. (US 2018/0348861 A1), as applied to claims 1-3, 7, 12, 16, 18 and 20, and further in view of Border et al. (US 2012/0120103 A1).
Regarding claim 19, Kuo in view of Castillo further in view of Uscinski teaches the computer system of claim 18, as described above.
Although Kuo in view of Castillo further in view of Uscinski teaches obtaining an image of the environment (Kuo, Para. [0042]), they do not explicitly teach “wherein the image is an overlaid image that includes content obtained from the image and a different image”. However, in an analogous field of endeavor, Border teaches the displayed image and the see-through view can be viewed as a combined image where one image is overlaid on the other or the two images can be simultaneously viewed in different portions of the see-through display that is viewable by the viewer (Border, Para. [0036]).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Kuo in view of Castillo further in view of Uscinski with the teachings of Border by including the image is an overlaid image containing content from an image and a different image. One having ordinary skill in the art would have been motivated to combine these references because doing so would allow for augmented image content aligned to objects form the surrounding environment, as recognized by Border. Thus, the claimed invention would have been obvious to one having ordinary skill in the art before the effective filing date.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Emma Rose Goebel whose telephone number is (703)756-5582. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571) 272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Emma Rose Goebel/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662