DETAILED ACTION
Claims 1, 3, and 5-14 are pending in this application, claims 1, 3, 5-9 and 11-14 have been amended. Claims 2 and 4 have been cancelled. Claims 1, 3, and 5-14 have been examined under the priority date 08/29/2022.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 08/28/2023 and 10/10/2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/04/2026 has been entered.
Response to Arguments
35 U.S.C. 112(b)
Applicant’s arguments (see Remarks, filed 05/04/2026) have been fully considered and are persuasive. In view of the amendments made to claim 1 the rejections made under 35 U.S.C. 112(b) have been withdrawn.
35 U.S.C. 101
Applicant’s arguments (see Remarks, filed 05/04/2026) regarding the rejections made under 35 U.S.C. 101 have been fully considered and are not persuasive. Taking claim 1 as example, claim 1 recites the following limitations which could each be reasonably classified as a mental process, abstract idea, or steps of mere data gathering;
“A detecting method executed by at least one processor, comprising: for each of a plurality of target objects: (Mental process of a detecting method which a human could reasonably perform by looking at an image and mentally “detecting” objects, where a processor is an additional element recited with a high level of generality, and does not meaningfully translate this step into practical application)
acquiring, for a captured image captured by a camera, depth information included in a plurality of pixels in the captured image wherein the depth information includes depth of a reference surface from the camera depth of the each of the plurality of the target objects from the camera Step of mere data gathering in which a human could collect an image, and acquire some depth information via a manual mathematical computation)
deriving a distance from the reference surface to a representative point of the each of the target objects a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information; (Mental process of looking at an image and assessing the distance of an object in an image via a manual math process)
identifying one or more of the plurality of target objects, from which the distance from the reference surface to the representative point of the one or more of the plurality of target objects is greater than or equal to a predetermined distance, as one or more candidate objects; (Mental process of looking at an image and assessing the distance of an object in an image and comparing the distance to a threshold distance manually)
and determining a gesture of one or more candidate objects, , one or more remaining target objects of the plurality of target objects(Mental process which could be performed by a human where a gesture is assessed visually).”
Under step 2A prong 1, the limitations recited are drawn to abstract ideas, mental processes or steps of mere data gathering as noted above by the examiner, the examiner has added further elaboration on why each step is drawn to a mental process or step of mere data gathering for clarity of record. Under step 2B prong 2, the claim recites the additional elements of “at least one processor” (claims 1, 13, and 14) and “non-transitory computer recording medium” (claim 14), which neither constitute judicial exception nor integrate claim into practical application. Under step 2B, the claim does not include elements or limitations that amount to significantly more than an abstract idea (See MPEP section 2106). Therefore, for at least the reasons stated above, the examiner maintains the rejections made under 35 U.S.C. 101. The examiner respectfully encourages the applicant to amend the claims to further clarify the components or analysis which perform the actions above to further translate the claim into practical application.
35 U.S.C. 103
Applicant’s arguments (see Remarks, filed 05/04/2026) regarding the rejections made to independent claims 1, 13 and 14 under 35 U.S.C. 1023 have been fully considered by the examiner and are not persuasive.
Applicant argues (see Remarks, filed 05/04/2026, page 9, paragraph 2) that the combination of Suzuki and Oshima would not render the claimed limitations of claims 1, 13 and 14 obvious because Suzuki fails to teach deriving, for multiple hands, a distance from a reference surface to a point on each of the hand along a direction within a predetermined range of normal to a reference surface. The examiner agrees that Suzuki does not teach this limitation and notes that Suzuki was not previously relied on to teach this limitation of claim 1.
Applicant further argues (see Remarks, filed 05/04/2026, page 9, paragraph 3) that Suzuki fails to teach identifying one or more hands, for which the distance from a reference surface to the representative point on the hand is greater than or equal to a predetermined distance, and determine that that hand is the candidate object. The examiner disagrees, and first notes that claim 1 does not recite the requirement that the target objects or candidate objects be limited to a plurality of hands, it merely recites that they be a plurality of objects, therefore the broadest reasonable interpretation of this limitation means there must only be more than one object of any kind. Further, Suzuki teaches in [0068] that 3D coordinates of feature points on the detected hand and multiple detected fingers are determined, where the fingers and hand are analogous to a plurality of target objects, and the feature points are analogous to a representative point. Suzuki then teaches in [0069] that the system determines a minimum Z coordinate, or distance from the camera as well as a distance from a 3D measurement area must be met in order for a hand or finger to be detected as a manipulating or gesturing object. These requirements would be analogous to a minimum distance threshold from the surface and a minimum distance threshold from the camera which must be met for the object to be detected as a manipulating object/candidate object.
PNG
media_image1.png
122
234
media_image1.png
Greyscale
PNG
media_image2.png
530
280
media_image2.png
Greyscale
(Suzuki, [0068]-[0069])
Applicant further argues (see Remarks, filed 05/04/2026, page 9, paragraph 3) that Suzuki fails to teach determining the gesture of one or more candidate objects, without determining the gestures of the remaining target objects. The examiner disagrees, the broadest reasonable interpretation of this limitation is that a candidate object or objects, which is an object meeting the limitations discussed above, have a gesture determined, and that the remaining objects in the image do not. This limitation does not require that all objects be of a single type, or that all objects be hands. In paragraphs [0066], [0069] and [0071] of Suzuki the gesture made by a hand and fingers is determined, and the coordinate positions of the objects/hands/fingers determines if a gesture may be determined or not, therefore if the positional requirements are not met, a gesture will not be determined for those objects. The plurality of fingers and the hand being analogous to the multiple objects which have been assessed for whether they meet the requirements for being a candidate object as discussed above. One of ordinary skill in the art would understand that not determining a gesture for the fingers or hand not meeting the requirements to be a candidate object will not have their gestures determined.
PNG
media_image3.png
178
278
media_image3.png
Greyscale
(Suzuki, [0066])
PNG
media_image4.png
168
282
media_image4.png
Greyscale
(Suzuki, [0069])
PNG
media_image5.png
224
280
media_image5.png
Greyscale
(Suzuki, [0071])
Applicant further argues (see Remarks, filed 05/04/2026, page 10, paragraph 2 through page 11 paragraph 1) that Oshima fails to teach “deriving a distance from the reference surface to a representative point of the each of the target objects along a direction within a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information” as recited by claim 1 in clean form. Applicant argues that Oshima cannot teach the limitation of deriving a distance from the reference surface, because it teaches determining fingertip touch points, therefore the distance from the surface is always zero. The examiner disagrees, the broadest reasonable interpretation of the limitation is a distance from the surface to a point on the object is determined, and that the distance be computed based on a direction which is within a threshold number of degrees of normal and uses the positional relationships between the camera, the surface and the object. Oshima teaches in [0098] and [0057] that the fingertip positions, which are interpreted as a target object, are measured as a vector distance from the flat surface, figures 6A and B show the fingertip distance determinations. These teachings of Oshima teach that the distance and three-dimensional positions of the fingertip are determined for when a fingertip is making contact with the surface and when it is not. Further, [0099] - [0100] of Oshima teaches that these vectors are orthogonal projections, where the position of the fingertip is orthogonal to the surface, and therefore the distance is measure along this perpendicular direction. The teachings of [0098] and [0100] of Oshima would be analogous to the deriving a distance from the reference surface to a representative point of the each of the target objects along a direction within a predetermined degrees of a normal to the reference surface as claimed in claim 1 since the distance is measured as an orthogonal vector to the tip of each finger. Further, Oshima teaches that the coordinate systems which the positional vectors are derived in are based on the positions of the camera, the reference plane and the objects, which one of ordinary skill in the art would understand means that all the derived distances in a coordinate system would be based on the positions of these objects as claimed in claim 1. Therefore, given the teachings of both Suzuki and Oshima, the examiner respectfully maintains the rejections made under 35 U.S.C. 103 for at least the reasons discussed above.
PNG
media_image6.png
292
284
media_image6.png
Greyscale
(Oshima, [0098])
PNG
media_image7.png
144
282
media_image7.png
Greyscale
(Oshima, [0099])
PNG
media_image8.png
118
284
media_image8.png
Greyscale
(Oshima, [0100])
PNG
media_image9.png
68
280
media_image9.png
Greyscale
PNG
media_image10.png
568
284
media_image10.png
Greyscale
(Oshima, [0032]-[0033])
PNG
media_image11.png
204
290
media_image11.png
Greyscale
(Oshima, [0057])
PNG
media_image12.png
250
530
media_image12.png
Greyscale
(Oshima, figure 6B)
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3, and 5-14 are rejected under U.S.C. 101 as being directed to an abstract idea without significantly more.
Regarding claims 1, 13 and 14, are directed to abstract ideas, mental processes or steps of mere data gathering. Specifically, the claims recite the limitations;
A detecting method executed by at least one processor, comprising: for each of a plurality of target objects:
acquiring, for a captured image captured by a camera, depth information included in a plurality of pixels in the captured image wherein the depth information includes depth of a reference surface from the camera depth of the each of the plurality of the target objects from the camera Mental process which could be performed by a human)
deriving a distance from the reference surface to a representative point of the each of the target objects a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information; (Mental process of looking at an image and assessing the distance of an object in an image))
identifying one or more of the plurality of target objects, from which the distance from the reference surface to the representative point of the one or more of the plurality of target objects is greater than or equal to a predetermined distance, as one or more candidate objects; (Mental process of looking at an image and assessing the distance of an object in an image)
and determining a gesture of one or more candidate objects, , one or more remaining target objects of the plurality of target objects(Mental process which could be performed by a human where a gesture is assessed visually).
Under step 2A prong 1, the limitations recited are drawn to abstract ideas, mental processes or steps of mere data gathering as noted above. Further, under step 2B prong 2, the claims recite the additional elements of “at least one processor” (claims 1, 13, and 14) and “non-transitory computer recording medium” (claim 14), which neither constitute judicial exception nor integrate claim into practical application. Under step 2B, the claim does not include elements that amount to significantly more than an abstract idea. See MPEP section 2106.
Dependent claims 3, and 5-12 do not add limitations that meaningfully translate the abstract idea into a practical application or add significantly more.
Regarding claim 3, the claim adds the limitations;
and extracting the each of the plurality of target objects in the captured image each of the plurality of target objects (Mental process of deriving a distance mathematically).
The limitations recited by claim 3 constitute a mental process or steps of mere data gathering without significantly more.
Regarding claim 5, the claim adds the limitations;
and wherein the identifying the one or more candidate objects comprises identifying one of the plurality of target objects, from which the distance from the reference surface to the representative point of the one or more of the plurality of targets is greater than or equal to the predetermined distance and is distances, as a candidate object. (Mental process of making a determination).
The limitations recited by claim 5 constitute a mental process without significantly more.
Regarding claim 6, the claim adds the limitations;
wherein the representative point of the each of the plurality of target objects is a centroid of a portion of the each of the plurality of target objects (Mental process of making a determination).
The limitations recited by claim 6 constitute a mental process without significantly more.
Regarding claim 7, the claim adds the limitations;
wherein each of the one or more candidate objects and reference surface one or more remaining target objects of the plurality of target objects. (Mental process of making a determination of image contents).
The limitations recited by claim 7 constitute a mental process without significantly more.
Regarding claim 8, the claim adds the limitations;
wherein each of the one or more candidate objects Mental process of making a determination of image contents).
The limitations recited by claim 8 constitute a mental process without significantly more.
Regarding claim 9, the claim adds the limitations;
wherein the reference surface Mental process of making a determination of image contents).
The limitations recited by claim 9 constitute a mental process without significantly more.
Regarding claim 10, the claim adds the limitations;
extracting a planar rectangular region with a constant depth or a continuously changing depth as the reference surface based on the depth information (Data gathering),
The limitations recited by claim 10 constitute a mental process or steps of mere data gathering without significantly more.
Regarding claim 11, the claim adds the limitations;
captured in the captured image. (Mental process of identifying image contents or position of an object in an image).
The limitations recited by claim 11 constitute a mental process or steps of mere data gathering without significantly more.
Regarding claim 12, the claim adds the limitations;
and identifying the reference surface based on a position of a sign in the captured image, the sign being at a predetermined position in an image displayed on the reference surface (Mental process of identifying image contents or position of an object in an image).
The limitations recited by claim 12 constitute a mental process or steps of mere data gathering without significantly more.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
2. Claims 1, 3, 6 and 8-14 are rejected under 35 U.S.C. 103 as being unpatentable over Suzuki (US 20140292723 A1) in view of Oshima (US 20160054859 A1).
Regarding claim 1 Suzuki teaches;
A detecting method executed by at least one processor, comprising (Suzuki, [0005] the device includes a processor):
for each of a plurality of target objects (Suzuki, [0039] and [0040] a camera captures images, and the depth and positions of a manipulating object, such as a hand and multiple fingers (target objects), as well as a manipulated object which is being manipulated or moved by the manipulating object (target objects)):
acquiring, for a captured image captured by a camera (Suzuki, [0039] cameras 1 and 2 captures images), depth information included in a plurality of pixels in the captured image wherein the depth information includes depth of a reference surface from the camera (Suzuki, [0043] cameras one and two are set up such that they are above a table surface (reference surface) and height of Lc are computed which is the distance from a camera to the table surface as shown in figure 2) and depth of the each of the plurality of the target objects from the camera (Suzuki, [0061] an image is captured including a hand, and two objects and then the images are processed, [0062]-[0065] coordinates for the pixels belonging to each objects are measures, the measuring unit measures the 3D coordinates of the manipulating object or objects, such as the hand and fingers (Detection of two or more target objects), coordinates may be detected for all finger tips (multiple reference points), as well as the objects being manipulated by the hand and fingers, stereoscopy is used to determine the coordinates, which means the positions of the objects in relation to the cameras are used [0066] positions of the hands and fingers in relation to the camera is detected to determine the 3D positions);
PNG
media_image13.png
364
482
media_image13.png
Greyscale
(Suzuki, figure 2, emphasis added)
[deriving a distance from the reference surface to a representative point of the each of the target objects a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information;]
identifying one or more of the plurality of target objects, from which the distance from the reference surface to the representative point of the one or more of the plurality of target objects is greater than or equal to a predetermined distance, as one or more candidate objects (Suzuki, [0068] the 3D positions of the coordinate feature point/points of the detected hand and fingers (plurality of target objects) is determined, and [0069] a minimum of the z coordinate (distance) from the cameras and the 3D Measurement enabled area (reference surface) must be met such that the hand may be detected, therefore in order for the hand or fingers to be determined as a manipulating object (candidate object) it must meet a detectable distance requirement);
and determining a gesture of one or more candidate objects, (Suzuki, [0066] the gesture is determined based on the shape of the hand and fingers and their change in shape between gestures, where the hand and fingertips are the candidate objects),
one or more remaining target objects of the plurality of target objects (Suzuki, [0069] and [0071] the coordinate positions of the target or candidate objects determines if a gesture may be determined, therefore if all 5 fingers on a hand do not fall within the distance parameters of detection only the ones meeting the requirements will have their gestures determined).
Suzuki does not teach; deriving a distance from the reference surface to a representative point of the each of the target objects a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information;
However, in the same field of endeavor, Oshima teaches deriving a distance from the reference surface to a representative point of the each of the target objects a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information (Oshima, [0098] and [0057] the fingertip positions (target objects) have a vector generated to measure the distances from the flat surface (reference surface) and [0100] vectors of the position are derived using orthogonal projections where the fingertip contact on the surface is an orthogonal vector, meaning the positional relationship/distance is derived as a distance perpendicular to the reference surface [0032] – [0033] the coordinate systems by which the positions are derived are based on the position of the camera and the position of the plane it is in view, therefore the distances of all the objects and surfaces relative to the camera are known);
The combination of Suzuki and Oshima would have obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Suzuki teaches a method of determining hand gestures and distances of the hand in relation to a flat surface, it does not teach that the target object or the hand is orthogonal to a flat surface. Oshima, teaches this limitation, and this feature would be advantageous to add to the system of Suzuki because determining hand and finger contact points and distance in relation to a flat surface requires reference points, and defining the plants as being perpendicular to one another allows the system to more accurately assess the vertical distance between the hand and the flat surface. (Oshima, [0032]- [0033] and [0054])
Regarding claim 3 the combination of Suzuki and Oshima teaches; The detecting method according to claim 1, further comprising:
and extracting the each of the plurality of target objects in the captured image each of the plurality of target objects (Suzuki, [0057] an image is captured of the hand (target or candidate object) and the image projected by the projector onto the flat surface (reference surface), as well as multiple objects on the surface (plurality of target objects), Figures 6A and 6B show the hand over the projector surface, where there is at least partial overlap),
PNG
media_image14.png
318
578
media_image14.png
Greyscale
(Suzuki, Figure 6A)
Regarding claim 6, the combination of Suzuki and Oshima teaches; The detecting method according to claim 1,
wherein the representative point of the each of the plurality of target objects is a centroid of a portion of the each of the plurality of target objects Suzuki, [0097] the projection of the image is based on the position of the hand (target object) and the border of the area (target area, this is the area the projector is projecting the image onto), Figure 12 and 13 A-C, show the position of the center of the hand (representatives point) being used, [0065] the center of the back of the hand may be used when determining the 3D coordinates of the hand, given that the center of the back of the hand can be used for the hand coordinates, this would mean the center of the hand (reference point) could be used to determine the distance from the projector surface, where figure 13a shows the center being determined for the hand when the whole hand overlaps the surface in the image).
Regarding claim 8, the combination of Suzuki and Oshima teaches;
The detecting method according to claim 1, wherein each of the one or more candidate objects (Suzuki, [0057] an image is captured of between the hand (candidate object), fingers (target or candidate objects) and the image projected by the projector surface or table (reference surface), Figure 1 shows the positional relationship of the hand and the surface, where the hand is located on a side of the surface).
PNG
media_image15.png
464
556
media_image15.png
Greyscale
(Suzuki, figure 1)
Regarding claim 9, the combination of Suzuki and Oshima teaches; The detecting method according to claim 1, wherein the reference surface (Suzuki, Figure 1, shows the surface as being a projection surface, [0005] projector displays and image on a surface).
Regarding claim 10, the combination of Suzuki and Oshima teaches; The detecting method according to claim 1, further comprising:
extracting a planar rectangular region with a constant depth or a continuously changing depth as the reference surface based on the depth information (Suzuki, [0039] the projection surface is a predetermined surface, [0043] this can be a table surface, which is a flat surface of consistent depth).
Regarding claim 11, the combination of Suzuki and Oshima teaches; The detecting method according to claim 1, further comprising:
and identifying the reference surface based on a position of a sign in the captured image, the sign being at a predetermined position of a component captured in the image (Suzuki, [0071] the projection surface is detected/displayed using a projected 3D border (sign) around the area which the coordinates and positions are known, analogous to the applicant’s definition of a sign being something placed on all 4 corners to detect the surface defined in [0036] of applicant’s specification).
Regarding claim 12, the combination of Suzuki and Oshima teaches; The detecting method according to claim 1, further comprising:
and identifying the reference surface based on a position of a sign in the captured image, the sign being at a predetermined position in an image displayed on the reference surface (Suzuki, [0071] the projection surface is detected/displayed using a 3D border around the area which the coordinates and positions are known, analogous to the applicant’s definition of a sign being something placed on all 4 corners to detect the surface defined in [0036] of applicant’s specification).
Regarding claim 13, the combination of Suzuki and Oshima teaches; A detecting device comprising at least one processor configured to (Suzuki, [0005] the device includes a processor):
for each of a plurality of target objects: acquire, from a captured image captured by a camera (Suzuki, [0039] and [0040] a camera captures images, and the depth and positions of a manipulating object, such as a hand and multiple fingers (target objects), as well as a manipulated object which is being manipulated or moved by the manipulating object (target objects)),
included in a plurality of pixels in the captured image, wherein the depth information includes depth of a reference surface from the camera (Suzuki, [0043] cameras one and two are set up such that they are above a table surface (reference surface) and height of Lc are computed which is the distance from a camera to the table surface as shown in figure 2) and depth of the each of the plurality of the target objects from the camera (Suzuki, [0061] an image is captured including a hand, and two objects and then the images are processed, [0062]-[0065] coordinates for the pixels belonging to each objects are measures, the measuring unit measures the 3D coordinates of the manipulating object or objects, such as the hand and fingers (Detection of two or more target objects), coordinates may be detected for all finger tips (multiple reference points), as well as the objects being manipulated by the hand and fingers, stereoscopy is used to determine the coordinates, which means the positions of the objects in relation to the cameras are used [0066] positions of the hands and fingers in relation to the camera is detected to determine the 3D positions);
and each of the target objects target object along a direction within a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information (Oshima, [0098] and [0057] the fingertip positions (target objects) have a vector generated to measure the distances from the flat surface (reference surface) and [0100] vectors of the position are derived using orthogonal projections where the fingertip contact on the surface is an orthogonal vector, meaning the positional relationship/distance is derived as a distance perpendicular to the reference surface [0032] – [0033] the coordinate systems by which the positions are derived are based on the position of the camera and the position of the plane it is in view, therefore the distances of all the objects and surfaces relative to the camera are known);
identify one or more of the plurality of target objects, from which the distance from the reference surface to the representative point of the one or more of the plurality of target objects is greater than or equal to a predetermined distance, as one or more candidate objects (Suzuki, [0068] the 3D positions of the coordinate feature point/points of the detected hand and fingers (plurality of target objects) is determined, and [0069] a minimum of the z coordinate (distance) from the cameras and the 3D Measurement enabled area (reference surface) must be met such that the hand may be detected, therefore in order for the hand or fingers to be determined as a manipulating object (candidate object) it must meet a detectable distance requirement); and
determine a gesture of the one or more candidate objects (Suzuki, [0066] the gesture is determined based on the shape of the hand and fingers and their change in shape between gestures, where the hand and fingertips are the candidate objects),
one or more remaining target objects of the plurality of target objects (Suzuki, [0069] and [0071] the coordinate positions of the target or candidate objects determines if a gesture may be determined, therefore if all 5 fingers on a hand do not fall within the distance parameters of detection only the ones meeting the requirements will have their gestures determined).
The combination of Suzuki and Oshima would have obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Suzuki teaches a method of determining hand gestures and distances of the hand in relation to a flat surface, it does not teach that the target object or the hand is orthogonal to a flat surface. Oshima, teaches this limitation, and this feature would be advantageous to add to the system of Suzuki because determining hand and finger contact points and distance in relation to a flat surface requires reference points, and defining the plants as being perpendicular to one another allows the system to more accurately assess the vertical distance between the hand and the flat surface. (Oshima, [0032]- [0033] and [0054])
Regarding claim 14, the combination of Suzuki and Oshima teaches; A non-transitory computer-readable recording medium storing a program that causes at least one processor to (Suzuki, [0005] the device includes a processor):
for each of a plurality of target objects (Suzuki, [0039] and [0040] a camera captures images, and the depth and positions of a manipulating object, such as a hand and multiple fingers (target objects), as well as a manipulated object which is being manipulated or moved by the manipulating object (target objects)):
acquire, from a captured image captured by a camera, depth information included in a plurality of pixels in the captured image, wherein the depth information includes depth of a reference surface from the camera (Suzuki, [0043] cameras one and two are set up such that they are above a table surface (reference surface) and height of Lc are computed which is the distance from a camera to the table surface as shown in figure 2) and depth of the each of the plurality of the target objects from the camera (Suzuki, [0061] an image is captured including a hand, and two objects and then the images are processed, [0062]-[0065] coordinates for the pixels belonging to each objects are measures, the measuring unit measures the 3D coordinates of the manipulating object or objects, such as the hand and fingers (Detection of two or more target objects), coordinates may be detected for all finger tips (multiple reference points), as well as the objects being manipulated by the hand and fingers, stereoscopy is used to determine the coordinates, which means the positions of the objects in relation to the cameras are used [0066] positions of the hands and fingers in relation to the camera is detected to determine the 3D positions);
derive a distance from the reference surface to a representative point of the each of the target objects a predetermined degrees of a normal to the reference surface based on the depth of the reference surface from the camera and the depth of the representative point of the each of the target objects from the camera included in the depth information (Oshima, [0098] and [0057] the fingertip positions (target objects) have a vector generated to measure the distances from the flat surface (reference surface) and [0100] vectors of the position are derived using orthogonal projections where the fingertip contact on the surface is an orthogonal vector, meaning the positional relationship/distance is derived as a distance perpendicular to the reference surface [0032] – [0033] the coordinate systems by which the positions are derived are based on the position of the camera and the position of the plane it is in view, therefore the distances of all the objects and surfaces relative to the camera are known);
identify one or more of the plurality of target objects, from which the distance from the reference surface to the representative point of the one or more of the plurality of target objects is greater than or equal to a predetermined distance, as one or more candidate objects (Suzuki, [0068] the 3D positions of the coordinate feature point/points of the detected hand and fingers (plurality of target objects) is determined, and [0069] a minimum of the z coordinate (distance) from the cameras and the 3D Measurement enabled area (reference surface) must be met such that the hand may be detected, therefore in order for the hand or fingers to be determined as a manipulating object (candidate object) it must meet a detectable distance requirement); and
determine a gesture of the one or more candidate objects, (Suzuki, [0066] the gesture is determined based on the shape of the hand and fingers and their change in shape between gestures, where the hand and fingertips are the candidate objects),
one or more remaining target objects of the plurality of target objects (Suzuki, [0069] and [0071] the coordinate positions of the target or candidate objects determines if a gesture may be determined, therefore if all 5 fingers on a hand do not fall within the distance parameters of detection only the ones meeting the requirements will have their gestures determined).
The combination of Suzuki and Oshima would have obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. Suzuki teaches a method of determining hand gestures and distances of the hand in relation to a flat surface, it does not teach that the target object or the hand is orthogonal to a flat surface. Oshima, teaches this limitation, and this feature would be advantageous to add to the system of Suzuki because determining hand and finger contact points and distance in relation to a flat surface requires reference points, and defining the plants as being perpendicular to one another allows the system to more accurately assess the vertical distance between the hand and the flat surface. (Oshima, [0032]- [0033] and [0054])
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Suzuki (US 20140292723 A1) in view of Oshima (US 20160054859 A1) and in further view of Xiong (US 20120062736 A1).
Regarding claim 5 the combination of Suzuki and Oshima fails to teach; The detecting method according to
wherein the identifying the one or more candidate objects comprises identifying one of the plurality of target objects, from which the distance from the reference surface to the representative point of the one or more of the plurality of targets is greater than or equal to the predetermined distance and is distances, as a candidate object.
However, in the same field of endeavor, Yoon teaches;
wherein the identifying the one or more candidate objects comprises identifying one of the plurality of target objects (Xiong, [0088] Figure 11 shows that multiple points are located in the image which correspond to portions of the hand/fingers (plurality of target objects)),
from which the distance from the reference surface to the representative point of the one or more of the plurality of targets is greater than or equal to the predetermined distance and is distances, as a candidate object (Xiong, [0089] the distances of the projection points (target objects) are determined from the center point of the palm surface (reference surface), [0090] where the furthest point from the surface is the fingertip/pointer (candidate object) which interacts with the computing system).
The combination of Suzuki, Oshima and Xiong would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation to add the depth mapping function in which an object is classified based on its position relative to a reference surface of Xiong is that this allows for detection of finger tips such that when the hand changes positions, the gestures which are used to interact with the system may be accurately determined. (Xiong, [0087]-[0091])
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Suzuki (US 20140292723 A1) in view of Oshima (US 20160054859 A1) and in further view of Kamamori (US 20170315674 A1).
Regarding claim 7, the combination of Suzuki and Oshima teaches; The detecting method according to claim 1, wherein each of the one or more candidate objects hand, and (Suzuki, [0039] the system detects the position of a manipulating object (candidate object) which is a hand or fingers and a manipulated object, such as a document being projected, [0040] the system determines the document has been touched, Figure 1 shows the user manipulating the document, which examiner is interpreting as the document being touch, moved or otherwise handled by the user),
The combination of Suzuki and Oshima fails to teach; wherein the reference surface one or more remaining target objects of the plurality of target objects.
However, in the same field of endeavor, Kamamori teaches; wherein the reference surface one or more remaining target objects of the plurality of target objects (Kamamori, [0068] the gripped target surface (reference surface) may be held by one hand (remaining target object) and interacted with by another hand (candidate object) and the two hands, the static hand and the moving hand, may be differentiated by the system [0079] the target surface which exists below the fingers or hand may be held by the hand or fingers, .
PNG
media_image16.png
220
538
media_image16.png
Greyscale
(Kamamori, Figure 4)
The combination of Suzuki, Oshima and Kamamori would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The combination of Suzuki and Oshima both teach methods of hand gesture detection with reference to a surface and multiple objects, but do not explicitly teach that a reference surface may be held. The motivation to add this method of Kamamori to the systems of Suzuki and Oshima is that detection of a held reference surface allows for the system to discriminate between gestures made by the hand not holding the surface such that erroneous interactive gestures which may be detected by the holding hand are suppressed. (Kamamori, [0067]-[0068])
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. For a listing of analogous prior art as cited by the examiner please see the attached PTO-892 Notice of References Cited.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN M ELLIOTT whose telephone number is (703)756-5463. The examiner can normally be reached M-F 8AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.E./Examiner, Art Unit 2666
/Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights