DETAILED ACTION
Claims 1-25 are pending in this application. Claims 1, 12, 13, 22 and 23 have been amended in this application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 03/28/2023 and 07/10/2023 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Priority
Claims 1-25 are given the priority date of 01/05/2022 in accordance with applicant’s provisional application 63/296,486.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/11/2026 has been entered.
Response to Arguments
35 U.S.C. 103
Applicant’s arguments (see Remarks filed 02/11/2026) regarding the previous claim rejections under 35 U.S.C. 103 in view of Abovitz, Hu and Wu have been fully considered by the examiner, and are persuasive. However, given the amendments to the scope of claims 1, 13 and 23, a new grounds of rejection is presented in view of Yun and Sivan as fully discussed below.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Stationary light transmittable display device in claims 1, 12, 13, 22, and 23
Depth sensing module of claims 1, 13 and 23
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-25 are rejected under 35 U.S.C. 103 as being unpatentable over Yun (US 20180217678 A1), in view of Sivan (US 20210034843 A1).
Regarding Claim 1 Yun discloses; An active interactive navigation system, comprising:
a stationary light-transmittable display device (Yun, [0005] there is a transparent display disposed between a user and objects, further there is no disclosure in the specification of the applicant regarding the displaying being “stationary” therefore the examiner is interpreting the disclosure of the multiple display types of displays in paragraph [0023] of the applicant’s specification as inherently having the capacity to be stationary or not based on application)
disposed between at least one user and a plurality of dynamic objects in a physical environment, wherein the at least one user views the dynamic objects through the stationary light-transmittable display device (Yun, [0005] a transparent display is placed between a user and multiple objects, [0148] the system may detect the movement of either the user or the objects, indicating the objects are dynamic, [0074]-[0075] the screen is positioned between a user and an object, where the user is able to view the object(s) through the screen as shown in figure 1);
PNG
media_image1.png
516
542
media_image1.png
Greyscale
(Yun, figure 1)
a target object image capturing device comprising a camera lens (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph of the object, [0012] where the sensor may be a camera which inherently has a lens, figure 2 shows the sensors which may be a camera/photographing unit as being coupled to the display), coupled to the stationary light-transmittable display device (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph, figure 2 shows the sensors which may be a camera/photographing unit as being coupled to the display),
configured to obtain a dynamic object image of the dynamic objects in the physical environment (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph of the object which is in a physical environment and may be moving, as described in [0210]-[0211]);
a user image capturing device, coupled to the stationary light- transmittable display device, configured to obtain a user image (Yun, [0348] the user may be directly photographed to determine characteristics);
and a processing device comprising a memory and a processor (Yun [0294] the system has a memory and processor), coupled to the stationary light- transmittable display device (Yun, [00294] the memory and the processor are coupled to the display device),
wherein the processor is configured to recognize the dynamic objects in the dynamic object image (Yun, [0101] the object type may be determined, [0257] there are a plurality of objects behind the transparent display and information corresponding to the objects may be displayed, [0258] object attributes and type information may be displayed for the objects, [0259] the object types may be determined),
track the dynamic objects (Yun, [0209] the objects position can be detected when new objects move in and out of the field of view, [0080] the positions of the objects can be detected, [0090]-[0091] the system has a sensor (which may be a camera per [0023]) detects the positions of the object which may be moving, which is analogous to object tracking, [0180] the object or users positions are detected, where the object or user may change position/move)
[recognize the at least one user and select a service user in the user image,
capture a facial feature of the service user,
determine whether the facial feature matches a plurality of facial feature points,
detect a line of sight of the service user if the facial feature matches the facial feature points,
perform image cutting to cut the user image into a plurality of images to be recognized if the facial feature does not match the facial feature points,
and perform user recognition on each of the images to be recognized to detect the line of sight of the service user,]
wherein the line of sight passes through the stationary light- transmittable display device to watch a target object among the dynamic objects (Yun, [0080] the system determines a line of sight of a user passing through a display apparatus to the object being looked at),
wherein the processor is further configured to recognize the target object watched by the service user according to the line of sight (Yun, [0086] the object being looked at may have its characteristics determined and displayed to the user [0101] the object type may be determined, [0257] there are a plurality of objects behind the transparent display and information corresponding to the objects may be displayed, [0258] object attributes and type information may be displayed for the objects, [0259] the object types may be determined),
[generate face position three-dimensional coordinates corresponding to the service user,]
generate position three-dimensional coordinates corresponding to the target object recognized in the dynamic object image (Yun, [0076] the positions of the user and the object may be determined [0183] a position of the user and the object may be detected in x and y and the a distance (third dimension) of the object and user may be determined using a depth camera as shown in figure 10),
PNG
media_image2.png
364
396
media_image2.png
Greyscale
(Yun, Figure 10)
and generate depth and width information of the target object wherein the depth information is detected by a depth sensing module of the target object image capturing device, (Yun, [0181] the distance information can be sensed using a depth camera or distance sensor for generating depth information for the target object, [0076] the attribute information of an object may be object size, which would include at least width and height of the object),
accordingly calculate a cross-point position on the stationary light-transmittable display device by computing a geometric intersection between a sight line vector extending from the service user toward the target object in the physical environment and a display plane of the stationary light- transmittable display device, based on the face position three-dimensional coordinates of the service user and the position three-dimensional coordinates of the target object (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, [0179] the point T(a), referenced as the transmissive area in the reference, is the points where the user’s line of sight directed to an object intersects with the display, [0222] the system may determine the user’s head and eye positions when the user views the apparatus (face coordinate), [0196] the transmissive area (point of intersection of the line of sight and the display) may be determined using the user’s gaze direction, position of the user and the position of the object),
and display virtual information corresponding to the target object on the cross-point position of the stationary light-transmittable display device (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, this is referenced in Yun as the transmissive area and is where the object is displayed on the display along with information about the object).
Yun does not disclose:
recognize the at least one user and select a service user in the user image,
capture a facial feature of the service user,
determine whether the facial feature matches a plurality of facial feature points,
detect a line of sight of the service user if the facial feature matches the facial feature points,
generate face position three-dimensional coordinates corresponding to the service user,
perform image cutting to cut the user image into a plurality of images to be recognized if the facial feature does not match the facial feature points,
and perform user recognition on each of the images to be recognized to detect the line of sight of the service user,
However, in the same field of endeavor Sivan teaches:
recognize the at least one user and select a service user in the user image (Sivan, [0007] multiple images are captured of a target person (service user) where the images are of at least one person, [0008] the target person is recognized (service user is recognized)),
capture a facial feature of the service user (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors),
determine whether the facial feature matches a plurality of facial feature points (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors, the features are then matched to other features to verify the target person’s identity),
detect a line of sight of the service user if the facial feature matches the facial feature points (Sivan, [0045]-[0046] during each iteration of facial recognition, the target person’s (service user) face is scanned and the line of sight is detected, as well as if the line of sight is obscured, [0044] the person’s face is iteratively captured and verified, so the line of sight is iteratively extracted after a verification step has occurred),
and perform user recognition on each of the images to be recognized to detect the line of sight of the service user (Sivan, [0045]-[0046] during each iteration of facial recognition, the target person’s (service user) face is scanned and the line of sight is detected, as well as if the line of sight is obscured, [0044] the person’s face is iteratively captured and verified, so the line of sight is iteratively extracted after a verification step has occurred, multiple rounds of iteration and verification and line of sight detection is performed),
generate face position three-dimensional coordinates corresponding to the service user (Sivan, [0046] the system uses the images to generate position and posture information about the target persons face, head and body),
perform image cutting to cut the user image into a plurality of images to be recognized if the facial feature does not match the facial feature points (Sivan, [0107] if the confidence score of recognition of the target person is not sufficient (features do not match), the drone may be repositioned, and multiple new target images may be captured and re-fed to the ML model, [0102] the ML model receives the images, and breaks them down into a plurality of features and labels (image cutting to break the image into multiple images/image segments)),
and perform user recognition on each of the images to be recognized to detect the line of sight of the service user (Sivan, [0102] the ML model receives the images, and breaks them down into a plurality of features and labels (image cutting to break the image into multiple images/image segments, [0088] as part of the facial recognition, line of sight may be determined for the user),
The combination of Yun and Sivan would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Yun teaches a transparent display for a which a user is detected and a target object is detected and recognized. Sivan teaches a method of facial recognition for user verification and line of sight detected. The addition of the facial verification method of Sivan would have improved the system of Yun by allowing user profiles to be created and verified for information access control purposes. (Sivan, [0002]-[0006])
Regarding claim 2 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 1, wherein the images to be recognized comprise a central image to be recognized and a plurality of peripheral images to be recognized (Yun, [0140] an image of an object may be captured (central image) and then a plurality of background images (peripheral images) may be captured to recognize object attributes and distance/depth measures about the object).
Regarding claim 3 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 1, wherein an overlapping region is present between one of the images to be recognized and another adjacent one (Yun, [0175] multiple images may be captured of the user or the object and they may be combined and displayed on the transparent display with an intermediate area of overlap, where the images of the user and the object are the two adjacent images, [0170] the areas in figure 7 displaying a value of “3” are regarded as areas in which displayed images are overlapped as shown in figure 7 below).
PNG
media_image3.png
362
526
media_image3.png
Greyscale
(Yun, Figure 7, emphasis added)
Regarding claim 4 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 3, wherein the one of the images to be recognized and the another adjacent one are vertically, horizontally, or diagonally adjacent to each other (Yun, [0175] multiple images may be captured of the user or the object and they may be combined and displayed on the transparent display with an intermediate area of overlap, where the images of the user and the object are the two adjacent images, [0170] the areas in figure 7 displaying a value of “3” are regarded as areas in which displayed images are overlapped as shown in figure 7 below, the areas shown as values of “1” and “2” correspond to the adjacent images being displayed on the screen ).
PNG
media_image3.png
362
526
media_image3.png
Greyscale
(Yun, Figure 7, emphasis added)
Regarding claim 5 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 1, wherein each of the target object image capturing device (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph of the object), the user image capturing device (Yun, [0348] the user may be directly photographed to determine characteristics), and the processor (Yun [0294] the system has a memory and processor) is programed by a program code to perform parallel computing separately and to perform parallel processing by using multi-threading with a multi-core central processing unit (Yun, [0296]-[0298] the system has multiple CPU and processor modules to execute different processes, [0302] the system may have commands/instructions stored in a memory and executed on the CPU, which the CPU is a multithread and multi core processor such that the tasks it is performing may be performed [0270] the display can adjust what is computed and displayed simultaneously, indicating parallel processing is occurring between the devices.
Regarding claim 6 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 1, wherein the user image capturing device comprises an RGB image sensing module (Yun, [0012] the system has a camera which captures images of the user and the object, [0086] the system may acquire and compare the color of the object or other characteristics indicating that the system has a color sensing module), a depth sensing module (Yun, [0181] the system has a depth image module), an inertial sensing module (Yun, [0314] the system has a navigation module to determine a map, location, route/travel information, which would be analogous to an inertial sensing module), and a GPS positioning sensing module (Yun, [0309] the system has a GPS module).
Regarding claim 7 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 1, wherein if the facial feature matches the facial feature points (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors, the features are then matched to other features to verify the target person’s identity),
the processor calculates a face position of the service user by using the facial feature points (Sivan, [0046] the system uses the images to generate position and posture information about the target persons face, head and body) and a line of sight direction of the line of sight ([0045]-[0046] during each iteration of facial recognition, the target person’s (service user) face is scanned and the line of sight is detected, as well as if the line of sight is obscured, [0044] the person’s face is iteratively captured and verified, so the line of sight is iteratively extracted after a verification step has occurred)
and generates a number corresponding to the service user (Sivan, [0012] the user’s face receives a face probability score (a number) generated in response to identification)
and the face position three-dimensional coordinates (Sivan, [0046] the system uses the images to generate position and posture information about the target persons face, head and body).
The combination of Yun and Sivan would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Yun teaches a transparent display for a which a user is detected and a target object is detected and recognized. Sivan teaches a method of facial recognition for user verification and line of sight detected. The addition of the facial verification method of Sivan would have improved the system of Yun by allowing user profiles to be created and verified for information access control purposes. (Sivan, [0002]-[0006])
Regarding claim 8 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 7, wherein the processor recognizes the at least one user in the user image (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors, the features are then matched to other features to verify the target person’s identity)
and selects the service user according to a service area range (Yun, [0080] a user is detected (service user) in front of the display [0082] the mapping area, shown as area 10’ in figure 1, is displayed once the user position is detected, the mapping area (service area) is generated once),
wherein the service area range has initial dimensions (Yun, [0082] the mapping area is estimated based on the dimensions of the user (initial dimensions).
PNG
media_image4.png
530
534
media_image4.png
Greyscale
(Yun Figure 1)
The combination of Yun and Sivan would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Yun teaches a transparent display for a which a user is detected in the area and a target object is detected and recognized. Sivan teaches a method of facial recognition for user verification and line of sight detected. The motivation for the combination is that the capacity to detect a service user based on range in Yun allows the interactive display to adjust according to user position to optimize user experience, while verification of user identity is helpful in potentially selecting display content based on user preferences. (Yun, [0274]-[0283] and Sivan, [0002]-[0006])
Regarding claim 9 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 8, wherein when the service user moves, the processor dynamically adjusts left and right dimensions of the service area range with the face position three-dimensional coordinates of the service user as a central point (Yun, [0082] the mapping area (service area) is estimated based on user dimensions and position, [0091] if the user’s position moves the display may update what is shown on the screen, which would include the mapping area, [0330]-[0331] the user’s face and line of sight are used to adjust the displayed content when the service user moved left or right, [0191] the transmissive area/mapping area/service area is updated dimensionally based on the user’s gaze, which is based on eye/face position and the transmissive area is adjusted to be displayed at the user’s eye/face level, since the area is adjusted based on the user’s eye level, it would be centered around the user’s face coordinates).
Regarding claim 10 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 9, wherein when the processor does not recognize the service user in the service area range in the user image, the processor resets the service area range to the initial dimensions (Yun, [0196] when the user moves or changes gaze direction, the transmissive/mapping/service area is updated, [0151] the area may be displayed/estimated if a user’s position is not detected, and then changed once a user is detected again, this area dimension when the user is not present is being interpreted as a standard dimension).
Regarding claim 11 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 1, further comprising: a database, coupled to the processor (Figure 28 and [0309] the database (DB) is connected to a memory and a part of a storage module which per [0302] is coupled to a CPU (processor)),
configured to store a plurality of object feature points corresponding to each one of the dynamic objects (Yun, [0287] the storage (which contains the database) is coupled to store characteristics (plurality of features) of products/objects),
wherein after the processor recognizes the target object watched by the service user (Yun, [0080] the system determines that a user is looking at an object through the screen, [0076] object attributes are determined (object recognition) and displayed),
the processor captures a pixel feature of the target object (Yun, [0138] object pixel information is captured)
and compares the pixel feature with the object feature points (Yun, [0141] object characteristics may be stored, then a detector may detect pixel characteristics in the image to compare or match the object to the stored characteristics/features),
and if the pixel feature matches the object feature points (Yun, [0141] object characteristics may be stored, then a detector may detect pixel characteristics in the image to compare or match the object to the stored characteristics/features, where the object is detected and determined/recognized based on pixel matching),
the processor generates a number corresponding to the target object (Yun, [0137]-[0138] the object pixels may be detected, where pixel values are assigned accordingly for the object (a number generated for the object)),
the position three-dimensional coordinates corresponding to the target object (Yun, [0076] object position in space relative to the display may be determined, [0316] coordinates for the position may be determined),
and the depth and width information of the target object (Yun, [0020] the object attributes may be determined, including [0022] object size, which would include depth and width information).
Regarding claim 12 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 1, wherein the processor determines whether the virtual information corresponding to the target object is superimposed and displayed on the cross-point position of the stationary light- transmittable display device (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, this is referenced in Yun as the transmissive area and is where the object is displayed on the display along with information about the object),
and if the virtual information is not superimposed nor displayed on the cross-point position of the stationary light- transmittable display device, the processor performs offset correction on a position of the virtual information (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, this is referenced in Yun as the transmissive area and is where the object is displayed on the display along with information about the object, where [0084] the information display area can be corrected based on the transmissive area point T (cross point) to such that the information is displayed in a favorable position).
Regarding claim 13 the combination of Yun and Sivan teaches; An active interactive navigation method adapted to an active interactive navigation system comprising a stationary light-transmittable display device (Yun, [0005] a transparent display is placed between a user and multiple objects, further there is no disclosure in the specification of the applicant regarding the displaying being “stationary” therefore the examiner is interpreting the disclosure of the multiple display types of displays in paragraph [0023] of the applicant’s specification as inherently having the capacity to be stationary or not based on application),
a target object image capturing device comprising a camera lens, (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph of the object, [0012] where the sensor may be a camera which inherently has a lens, figure 2 shows the sensors which may be a camera/photographing unit as being coupled to the display),
a user image capturing device comprising a camera lens (Yun, [0348] the user may be directly photographed to determine characteristics using a camera or sensor),
and a processing device comprising a memory and a processor (Yun [0294] the system has a memory and processor)),
wherein the stationary light transmittable display device is disposed between at least one user and a plurality of dynamic objects in a physical environment, wherein the at least one user views the dynamic objects through the stationary light-transmittable display device (Yun, [0005] a transparent display is placed between a user and multiple objects, [0148] the system may detect the movement of either the user or the objects, indicating the objects are dynamic, [0074]-[0075] the screen is positioned between a user and an object, where the user is able to view the object(s) through the screen as shown in figure 1);
the processor is configured to execute the active interactive navigation method (Yun [0294] the system has a memory and processor to execute code),
and the active interactive navigation method comprises: capturing, by the target object image capturing device, a dynamic object image of the dynamic objects in the physical environment (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph of the object which is in a physical environment and may be moving, as described in [0210]-[0211]),
recognizing the dynamic objects in the dynamic object image (Yun, [0101] the object type may be determined, [0257] there are a plurality of objects behind the transparent display and information corresponding to the objects may be displayed, [0258] object attributes and type information may be displayed for the objects, [0259] the object types may be determined),
and tracking the dynamic objects (Yun, [0209] the objects position can be detected when new objects move in and out of the field of view, [0080] the positions of the objects can be detected, [0090]-[0091] the system has a sensor (which may be a camera per [0023]) detects the positions of the object which may be moving, which is analogous to object tracking, [0180] the object or users positions are detected, where the object or user may change position/move);
obtaining, by the user image capturing device, a user image (Yun, [0348] the user may be directly photographed to determine characteristics),
recognizing the at least one user in the user image and selecting a service user (Sivan, [0007] multiple images are captured of a target person (service user) where the images are of at least one person, [0008] the target person is recognized (service user is recognized)),
capturing a facial feature of the service user and determining whether the facial feature matches a plurality of facial feature points (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors, the features are then matched to other features to verify the target person’s identity),
detecting a line of sight of the service user if the facial feature matches the facial feature points (Sivan, [0045]-[0046] during each iteration of facial recognition, the target person’s (service user) face is scanned and the line of sight is detected, as well as if the line of sight is obscured, [0044] the person’s face is iteratively captured and verified, so the line of sight is iteratively extracted after a verification step has occurred),
performing image cutting to cut the user image into a plurality of images to be recognized if the facial feature does not match the facial feature points (Sivan, [0107] if the confidence score of recognition of the target person is not sufficient (features do not match), the drone may be repositioned, and multiple new target images may be captured and re-fed to the ML model, [0102] the ML model receives the images, and breaks them down into a plurality of features and labels (image cutting to break the image into multiple images/image segments),
and performing user recognition on each of the images to be recognized to detect the line of sight of the service user (Sivan, [0102] the ML model receives the images, and breaks them down into a plurality of features and labels (image cutting to break the image into multiple images/image segments, [0088] as part of the facial recognition, line of sight may be determined for the user),
wherein the line of sight passes through the stationary light-transmittable display device to watch a target object among the dynamic objects in a physical environment (Yun, [0080] the system determines a line of sight of a user passing through a display apparatus to the object being looked at);
and recognizing the target object watched by the service user according to the line of sight (Yun, [0086] the object being looked at may have its characteristics determined and displayed to the user [0101] the object type may be determined, [0257] there are a plurality of objects behind the transparent display and information corresponding to the objects may be displayed, [0258] object attributes and type information may be displayed for the objects, [0259] the object types may be determined),
generating face position three-dimensional coordinates corresponding to the service user (Sivan, [0046] the system uses the images to generate position and posture information about the target persons face, head and body),
generating position three-dimensional coordinates corresponding to the target object recognized in the dynamic object image (Yun, [0076] the positions of the user and the object may be determined [0183] a position of the user and the object may be detected in x and y and the a distance (third dimension) of the object and user may be determined using a depth camera as shown in figure 10),
and generating depth and width information of the target object wherein the depth information is detected by a depth sensing module of the target object image capturing device, (Yun, [0181] the distance information can be sensed using a depth camera or distance sensor for generating depth information for the target object, [0076] the attribute information of an object may be object size, which would include at least width and height of the object),
accordingly calculating a cross-point position on the stationary light-transmittable display device by computing a geometric intersection between a sight line vector extending from the service user toward the target object in the physical environment and a display plane of the stationary light- transmittable display device, based on the face position three-dimensional coordinates of the service user and the position three-dimensional coordinates of the target object (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, [0179] the point T(a), referenced as the transmissive area in the reference, is the points where the user’s line of sight directed to an object intersects with the display, [0222] the system may determine the user’s head and eye positions when the user views the apparatus (face coordinate), [0196] the transmissive area (point of intersection of the line of sight and the display) may be determined using the user’s gaze direction, position of the user and the position of the object),
and displaying virtual information corresponding to the target object on the cross-point position of the stationary light-transmittable display device (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, this is referenced in Yun as the transmissive area and is where the object is displayed on the display along with information about the object).
The combination of Yun and Sivan would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Yun teaches a transparent display for a which a user is detected and an target object is detected and recognized. Sivan teaches a method of facial recognition for user verification and line of sight detected. The addition of the facial verification method of Sivan would have improved the system of Yun by allowing user profiles to be created and verified for information access control purposes. (Sivan, [0002]-[0006])
Regarding claim 14 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 13, wherein the images to be recognized comprise a central image to be recognized and a plurality of peripheral images to be recognized (Yun, [0140] an image of an object may be captured (central image) and then a plurality of background images (peripheral images) may be captured to recognize object attributes and distance/depth measures about the object).
Regarding claim 15 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 13, wherein an overlapping region is present between one of the images to be recognized and another adjacent one (Yun, [0175] multiple images may be captured of the user or the object and they may be combined and displayed on the transparent display with an intermediate area of overlap, where the images of the user and the object are the two adjacent images, [0170] the areas in figure 7 displaying a value of “3” are regarded as areas in which displayed images are overlapped as shown in figure 7 below).
Regarding claim 16 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 15, wherein the one of the images to be recognized and the another adjacent one are vertically, horizontally, or diagonally adjacent to each other (Yun, [0175] multiple images may be captured of the user or the object and they may be combined and displayed on the transparent display with an intermediate area of overlap, where the images of the user and the object are the two adjacent images, [0170] the areas in figure 7 displaying a value of “3” are regarded as areas in which displayed images are overlapped as shown in figure 7 below, the areas shown as values of “1” and “2” correspond to the adjacent images being displayed on the screen).
Regarding claim 17 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 13, further comprising: calculating a face position of the service user by using the facial feature points and a line of sight direction of the line of sight if the facial feature matches the facial feature points (Sivan, [0046] the system uses the images to generate position and posture information about the target persons face, head and body) and a line of sight direction of the line of sight ([0045]-[0046] during each iteration of facial recognition, the target person’s (service user) face is scanned and the line of sight is detected, as well as if the line of sight is obscured, [0044] the person’s face is iteratively captured and verified, so the line of sight is iteratively extracted after a verification step has occurred);
and generating a number and the face position three-dimensional coordinates corresponding to the service user (Sivan, [0012] the user’s face receives a face probability score (a number) generated in response to identification).
The combination of Yun and Sivan would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Yun teaches a transparent display for a which a user is detected and a target object is detected and recognized. Sivan teaches a method of facial recognition for user verification and line of sight detected. The addition of the facial verification method of Sivan would have improved the system of Yun by allowing user profiles to be created and verified for information access control purposes. (Sivan, [0002]-[0006])
Regarding claim 18 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 17, wherein the step of obtaining, by the user image capturing device, the user image and recognizing the at least one user in the user image and selecting the service user further comprises (Yun, [0348] the user may be directly photographed to determine characteristics):
recognizing the at least one user in the user image (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors, the features are then matched to other features to verify the target person’s identity) and selecting the service user according to a service area range (Yun, [0080] a user is detected (service user) in front of the display [0082] the mapping area, shown as area 10’ in figure 1, is displayed once the user position is detected, the mapping area (service area) is generated once), wherein the service area range has initial dimensions (Yun, [0082] the mapping area is estimated based on the dimensions of the user (initial dimensions).
Regarding claim 19 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 18, further comprising: dynamically adjusting left and right dimensions of the service area range with the face position three-dimensional coordinates of the service user as a central point when the service user moves (Yun, [0082] the mapping area (service area) is estimated based on user dimensions and position, [0091] if the user’s position moves the display may update what is shown on the screen, which would include the mapping area, [0330]-[0331] the user’s face and line of sight are used to adjust the displayed content when the service user moved left or right, [0191] the transmissive area/mapping area/service area is updated dimensionally based on the user’s gaze, which is based on eye/face position and the transmissive area is adjusted to be displayed at the user’s eye/face level, since the area is adjusted based on the user’s eye level, it would be centered around the user’s face coordinates).
Regarding claim 20 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 19, further comprising: resetting the service area range to the initial dimensions when the service user is not recognized in the service area range in the user image (Yun, [0196] when the user moves or changes gaze direction, the transmissive/mapping/service area is updated, [0151] the area may be displayed/estimated if a user’s position is not detected, and then changed once a user is detected again, this area dimension when the user is not present is being interpreted as a standard dimension).
Regarding claim 21 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 13, further comprising:
capturing a pixel feature of the target object and comparing the pixel feature with the object feature points (Yun, [0138] object pixel information is captured, [0141] object characteristics may be stored, then a detector may detect pixel characteristics in the image to compare or match the object to the stored characteristics/features);
and generating a number corresponding to the target object (Yun, [0137]-[0138] the object pixels may be detected, where pixel values are assigned accordingly for the object (a number generated for the object)), the position three-dimensional coordinates corresponding to the target object (Yun, [0076] object position in space relative to the display may be determined, [0316] coordinates for the position may be determined), and the depth and width information of the target object (Yun, [0020] the object attributes may be determined, including [0022] object size, which would include depth and width information).
Regarding claim 22 the combination of Yun and Sivan teaches; The active interactive navigation method according to claim 13, further comprising:
determining whether the virtual information corresponding to the target object is superimposed and displayed on the cross-point position of the stationary light- transmittable display device (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, this is referenced in Yun as the transmissive area and is where the object is displayed on the display along with information about the object),
and if the virtual information is not superimposed nor displayed on the cross-point position of the stationary light- transmittable display device, the processor performs offset correction on a position of the virtual information (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, this is referenced in Yun as the transmissive area and is where the object is displayed on the display along with information about the object, where [0084] the information display area can be corrected based on the transmissive area point T (cross point) to such that the information is displayed in a favorable position).
Regarding claim 23 the combination of Yun and Sivan teaches; An active interactive navigation system, comprising:
a stationary light-transmittable display device (Yun, [0005] there is a transparent display disposed between a user and objects, further there is no disclosure in the specification of the applicant regarding the displaying being “stationary” therefore the examiner is interpreting the disclosure of the multiple display types of displays in paragraph [0023] of the applicant’s specification as inherently having the capacity to be stationary or not based on application)
disposed between at least one user and a plurality of dynamic objects in a physical environment, wherein the at least one user views the dynamic objects through the stationary light-transmittable display device (Yun, [0005] a transparent display is placed between a user and multiple objects, [0148] the system may detect the movement of either the user or the objects, indicating the objects are dynamic, [0074]-[0075] the screen is positioned between a user and an object, where the user is able to view the object(s) through the screen as shown in figure 1);
PNG
media_image1.png
516
542
media_image1.png
Greyscale
(Yun, figure 1)
a target object image capturing device comprising a camera lens (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph of the object, [0012] where the sensor may be a camera which inherently has a lens, figure 2 shows the sensors which may be a camera/photographing unit as being coupled to the display), coupled to the stationary light-transmittable display device (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph, figure 2 shows the sensors which may be a camera/photographing unit as being coupled to the display),
configured to obtain a dynamic object image of the dynamic objects in the physical environment (Yun, [0023] the system may have at least one sensor which may be a photographing unit to capture a photograph of the object which is in a physical environment and may be moving, as described in [0210]-[0211]);
a user image capturing device, coupled to the stationary light- transmittable display device, configured to obtain a user image (Yun, [0348] the user may be directly photographed to determine characteristics);
and a processing device comprising a memory and a processor (Yun [0294] the system has a memory and processor), coupled to the stationary light- transmittable display device (Yun, [00294] the memory and the processor are coupled to the display device),
wherein the processor is configured to recognize the dynamic objects in the dynamic object image (Yun, [0101] the object type may be determined, [0257] there are a plurality of objects behind the transparent display and information corresponding to the objects may be displayed, [0258] object attributes and type information may be displayed for the objects, [0259] the object types may be determined),
track the dynamic objects (track)
recognize the at least one user and select a service user in the user image (Sivan, [0007] multiple images are captured of a target person (service user) where the images are of at least one person, [0008] the target person is recognized (service user is recognized)),
capture a facial feature of the service user (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors),
determine whether the facial feature matches a plurality of facial feature points (Sivan, [0102] the facial image is sent to a machine learning algorithm to extract multiple facial feature vectors, the features are then matched to other features to verify the target person’s identity),
detect a line of sight of the service user if the facial feature matches the facial feature points (Sivan, [0045]-[0046] during each iteration of facial recognition, the target person’s (service user) face is scanned and the line of sight is detected, as well as if the line of sight is obscured, [0044] the person’s face is iteratively captured and verified, so the line of sight is iteratively extracted after a verification step has occurred),
and perform user recognition on each of the images to be recognized to detect the line of sight of the service user (Sivan, [0045]-[0046] during each iteration of facial recognition, the target person’s (service user) face is scanned and the line of sight is detected, as well as if the line of sight is obscured, [0044] the person’s face is iteratively captured and verified, so the line of sight is iteratively extracted after a verification step has occurred, multiple rounds of iteration and verification and line of sight detection is performed),
generate face position three-dimensional coordinates corresponding to the service user (Sivan, [0046] the system uses the images to generate position and posture information about the target persons face, head and body),
perform image cutting to cut the user image into a plurality of images to be recognized if the facial feature does not match the facial feature points (Sivan, [0107] if the confidence score of recognition of the target person is not sufficient (features do not match), the drone may be repositioned, and multiple new target images may be captured and re-fed to the ML model, [0102] the ML model receives the images, and breaks them down into a plurality of features and labels (image cutting to break the image into multiple images/image segments)),
and perform user recognition on each of the images to be recognized to detect the line of sight of the service user (Sivan, [0102] the ML model receives the images, and breaks them down into a plurality of features and labels (image cutting to break the image into multiple images/image segments, [0088] as part of the facial recognition, line of sight may be determined for the user),
wherein the line of sight passes through the stationary light- transmittable display device to watch a target object among the dynamic objects (Yun, [0080] the system determines a line of sight of a user passing through a display apparatus to the object being looked at),
wherein the processor is further configured to recognize the target object watched by the service user according to the line of sight (Yun, [0086] the object being looked at may have its characteristics determined and displayed to the user [0101] the object type may be determined, [0257] there are a plurality of objects behind the transparent display and information corresponding to the objects may be displayed, [0258] object attributes and type information may be displayed for the objects, [0259] the object types may be determined),
generate face position three-dimensional coordinates corresponding to the service user (Sivan, [0046] the system uses the images to generate position and posture information about the target persons face, head and body)
generate position three-dimensional coordinates corresponding to the target object recognized in the dynamic object image (Yun, [0076] the positions of the user and the object may be determined [0183] a position of the user and the object may be detected in x and y and the a distance (third dimension) of the object and user may be determined using a depth camera as shown in figure 10),
PNG
media_image2.png
364
396
media_image2.png
Greyscale
(Yun, Figure 10)
and generate depth and width information of the target object wherein the depth information is detected by a depth sensing module of the target object image capturing device, (Yun, [0181] the distance information can be sensed using a depth camera or distance sensor for generating depth information for the target object, [0076] the attribute information of an object may be object size, which would include at least width and height of the object),
accordingly calculate a cross-point position on the stationary light-transmittable display device by computing a geometric intersection between a sight line vector extending from the service user toward the target object in the physical environment and a display plane of the stationary light- transmittable display device, based on the face position three-dimensional coordinates of the service user and the position three-dimensional coordinates of the target object (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, [0179] the point T(a), referenced as the transmissive area in the reference, is the points where the user’s line of sight directed to an object intersects with the display, [0222] the system may determine the user’s head and eye positions when the user views the apparatus (face coordinate), [0196] the transmissive area (point of intersection of the line of sight and the display) may be determined using the user’s gaze direction, position of the user and the position of the object),
and display virtual information corresponding to the target object on the cross-point position of the stationary light-transmittable display device (Yun, [0184] the person’s line of sight and object positions may be used to compute a point T which is the point where the person’s line of sight crosses the transparent display to see the object point T is shown in figure 10, this is referenced in Yun as the transmissive area and is where the object is displayed on the display along with information about the object).
The combination of Yun and Sivan would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Yun teaches a transparent display for a which a user is detected and a target object is detected and recognized. Sivan teaches a method of facial recognition for user verification and line of sight detected. The addition of the facial verification method of Sivan would have improved the system of Yun by allowing user profiles to be created and verified for information access control purposes. (Sivan, [0002]-[0006])
Regarding claim 24 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 23, wherein when the service user moves, the processor dynamically adjusts left and right dimensions of the service area range with the face position three-dimensional coordinates of the service user as a central point (Yun, [0082] the mapping area (service area) is estimated based on user dimensions and position, [0091] if the user’s position moves the display may update what is shown on the screen, which would include the mapping area, [0330]-[0331] the user’s face and line of sight are used to adjust the displayed content when the service user moved left or right, [0191] the transmissive area/mapping area/service area is updated dimensionally based on the user’s gaze, which is based on eye/face position and the transmissive area is adjusted to be displayed at the user’s eye/face level, since the area is adjusted based on the user’s eye level, it would be centered around the user’s face coordinates).
Regarding claim 25 the combination of Yun and Sivan teaches; The active interactive navigation system according to claim 24, wherein when the processor does not recognize the service user in the service area range in the user image, the processor resets the service area range to the initial dimensions (Yun, [0196] when the user moves or changes gaze direction, the transmissive/mapping/service area is updated, [0151] the area may be displayed/estimated if a user’s position is not detected, and then changed once a user is detected again, this area dimension when the user is not present is being interpreted as a standard dimension).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Salter (US 20140368533 A1) teaches a method of augmented reality displaying virtual object information to the user.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN M ELLIOTT whose telephone number is (703)756-5463. The examiner can normally be reached M-F 8AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.E./Examiner, Art Unit 2666 /EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666