DETAILED ACTION
This office action is responsive to the request for response filed 12/5/2025. The application contains claims 1-7, 9-10,12-17, all examined and rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-7, 9-10, 12-13, 15-17 rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Independent claims require to initiate a movement of a mobile object based on a combination of sound and gesture commands. Examiner could not find support for such combination. The specification disclose the ability to process commands based on gesture, sound, touch panel, OR terminal input. For example the specification of the published application ¶66 disclose that commands may come from gesture or sound “based on gesture or sound of a user” and not both combines, ¶91-97 disclose registration intention based on three independent alternatives gesture OR sound OR touch not a combination, and finally ¶¶140-148 show the ability to use gestures for control without the usage of any sound commands. The dependent claims inherit the deficiencies of the independent claims.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16 is rejected under pre-AIA 35 U.S.C. § 112, second paragraph, as being indefinite. Claims 16 recites the limitation " the second camera ". There is insufficient antecedent basis for this limitation in the claims.
Claims 17 is rejected under pre-AIA 35 U.S.C. § 112, second paragraph, as being indefinite. Claims 17 recites the limitation " the second camera ". There is insufficient antecedent basis for this limitation in the claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7, 9-10, 12-17 are rejected under 35 U.S.C. § 101 because the instant application is directed to non-patentable subject matter. Specifically, the claims are directed toward at least one judicial exception without reciting additional elements that amount to significantly more than the judicial exception. The rationale for this determination is in accordance with the guidelines of USPTO, applies to all statutory categories, and is explained in detail below.
When considering subject matter eligibility under 35 U.S.C. 101, (1) it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter. If the claim does fall within one of the statutory categories, (2a) it must then be determined whether the claim is directed to a judicial exception (i.e., law of nature, natural phenomenon, and abstract idea), and if so (2b), it must additionally be determined whether the claim is a patent-eligible application of the exception. If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim amounts to significantly more than the abstract idea itself. Examples of abstract ideas include certain methods of organizing human activities; a mental processes; and mathematical concepts, (2019 PEG)
STEP 1.
Per Step 1 of the two-step analysis, the claims are determined to be directed to a statutory eligibility category.
Step 2A: Prong One:
The invention is directed to recognize user’s gestures based on the user gestures and region or distance within captured images which is akin to Mental Process (see Alice), As such, the claims include an abstract idea. When considering the limitations individually and as a whole the limitations directed to the abstract idea are:
recognize a region where the user is present when the image is captured (Mental process, observation, evaluation, and judgment); in a case in which the user is present in a first region when the image is captured, recognize a gesture of the user on the basis of the image and first information for recognizing the gesture of the user (Mental process, observation and judgment); wherein the gesture of the user and a defined sound generated by the user initiates a movement of a mobile object (Mental process, observation and judgment), in a case in which the user is present in a second region when the image is captured, recognize a gesture of the user on the basis of the image and second information for recognizing the gesture of the user (Mental process, observation and judgment);
in response to recognizing the gesture and the defined sound, determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point, wherein the trajectory between the first trajectory point and the second trajectory point is determined (Mental process, observation and judgment),
trajectory is further determined by combining an arc of a group of arcs representative of the trajectory, wherein the arc of the group of arcs has a different curvature radii from other arcs comprising the group of arcs, wherein the arc of the group arcs is associated with a predicted time period of a sequence of predicted time periods, and wherein the predicted time period indicates a defined position on the trajectory along which the mobile object moves (Mental process, observation and judgment),
trajectory determined based on the conversion of coordinate representations of the first trajectory point and the second trajectory point (Mental process, observation and judgment)
Step 2A: Prong Two:
When considering the additional elements they do not integrate the judicial exception into a practical application. The additional elements are:
a storage device configured to store instructions; and one or more processors, wherein the one or more processors execute the instructions stored in the storage device (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2)), acquire an image capturing a user (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2)), wherein the image is pixel data representative of camera coordinate system data (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use. It does not integrate the abstract idea into a practical application);
initiating the movement the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory and the second trajectory device (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2) in addition it could fall also as an (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands),
based on the trajectory determined based on the conversion of coordinate representations, causing the mobile object to move along the trajectory from the first trajectory point to the second trajectory point (insignificant extra-solution activity, MPEP 2106.05(g)).
This judicial exception is not integrated into a practical application. The elements are recited at a high level of generality, i.e. a generic computing system performing generic functions including generic processing of data. Accordingly the additional elements do not integrate the abstract into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Therefore the claims are directed to an abstract idea. (2019 Revised Patent Subject Matter Eligibility Guidance ("2019 PEG"). Thus, under Step 2A of the Mayo framework, the Examiner holds that the claims are directed to concepts identified as abstract.
STEP 2B.
Because the claims include one or more abstract ideas, the examiner now proceeds to Step 2B of the analysis, in which the examiner considers if the claims include individually or as an ordered combination limitations that are "significantly more" than the abstract idea itself. This includes analysis as to whether there is an improvement to either the "computer itself," "another technology," the "technical field," or significantly more than what is "well-understood, routine, or conventional" in the related arts.
The instant application includes in Claim 1 additional steps to those deemed to be abstract idea(s).
When taken the steps individually, these steps are:
a storage device configured to store instructions; and one or more processors, wherein the one or more processors execute the instructions stored in the storage device (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2)), acquire an image capturing a user (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2)),
acquire an image capturing a user (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2)), wherein the image is pixel data representative of camera coordinate system data (description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use. It does not integrate the abstract idea into a practical application, MPEP 2106.05(h));
initiating the movement of the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory and the second trajectory device (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2) in addition it could fall also as an (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)), based on the trajectory determined based on the conversion of coordinate representations, causing the mobile object to move along the trajectory from the first trajectory point to the second trajectory point, is Well-Understood-Routine, Conventional activity as shown by Savarit et al. [US 2018/0012502 A1] See at least Fig. 2, ¶31, “aircraft is slaved to the calculated trajectory which is frozen”, ¶20, “FMS can, on instruction from the pilot, slave the aircraft automatically to the calculated trajectory”, Wakisaka et al. [US 20210154836 A1] See at least Fig. 9, ¶¶101-102, “trajectory control device 10 of the embodiment can obtain the amount of movement Ax of the X coordinate of the reference point P1, the amount of movement Δy of the Y coordinate of the reference point P1, and the gradient angle Δθ of a workpiece side surface after movement with respect to the workpiece side surface in the reference position, by means of a simple detection control using the contact sensor 26, without using a high-precision camera or image sensor, and can correct the trajectory in real time on the basis of these pieces of data”, Cord et al. [US 20220326718 A1] See at least ¶¶64-67, ¶68, “Go to the battery recharging location; task commanding the robot to move toward the battery recharging area. The attributes of this task are for example the coordinates of the starting point (implicitly the current coordinates of the robot) and the coordinates of the end of the movement trajectory (location of the recharging point), or the succession of movement segments”, Lurz et al. [US 20090271035 A1] See at least ¶8, ¶20, “invention also relates to a device for the computer-aided movement planning of a robot, with a trajectory for the movement of a spatial point assigned to the robot being planned during operation of the device in a stationary coordinates system”.
In the instant case, Claim 1 is directed to above mentioned abstract idea. Technical functions such as sending, receiving, displaying and processing data are common and basic functions in computer technology. The individual limitations are recited at a high level and do not provide any specific technology or techniques to perform the functions claimed.
Looking to MPEP 2106.05 (d), based on court decisions well understood, routine and conventional computer functions or mere instruction and/or insignificant activity have been identified to include: Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321,120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TU Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); O/P Techs., /no., v. Amazon.com, Inc., 788 F,3d 1359, 1363, 115 USPQ2d 1090,1093 (Fed. Cir, 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPG2d 1097, 1106 (Fed. Cir. 2014) ("Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result-a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink," (emphasis added)}; Insignificant intermediate or post solution activity -See Bilski v. Kappos, 581 U.S. 593, 611 -12, 95 USPQ2d 1001,1010 (2010) (well-known random analysis techniques to establish the inputs of an equation were token extra-solution activity); In Bilski referring to Flook, where Flook determined that an insignificant post-solution activity does not makes an otherwise patent ineligible claim patent eligible. In Bilski, the court added to Flook that pre-solution (such as data gathering) and insignificant step in the middle of a process (such as receiving user input) to be equally ineffective. The specification and Claim does not provide any specific process with respect to the display output that would transform the function beyond what is well understood. Like as found in Electric Power Group, Bilski, the technical process to implement the input and display functions are conventional and well understood.
In addition, when the claims are taken as a whole, as an ordered combination, the combination of steps does not add "significantly more" by virtue of considering the steps as a whole, as an ordered combination. The instant application, therefore, still appears only to implement the abstract idea to the particular technological environments using what is well-understood, routine, and conventional in the related arts. The steps are still a combination made to the abstract idea. The additional steps only add to those abstract ideas using well-understood and conventional functions, and the claims do not show improved ways of, for example, an unconventional non-routine functions for authorizing the timing of a payment and to activate a display screen based on a trigger or camera functions that could then be pointed to as being "significantly more" than the abstract ideas themselves. Moreover, Examiner was not able to identify any "unconventional" steps, which, when considered in the ordered combination with the other steps, could have transformed the nature of the abstract idea previously identified. The instant application, therefore, still appears to only implement the abstract ideas to the particular technological environments using what is well-understood, routine, and conventional in the related arts.
Further, note that the limitations, in the instant claims, are done by the generically recited computing devices. The limitations are merely instructions to implement the abstract idea on a computing device that is recited in an abstract level and require no more than a generic computing devices to perform generic functions.
CONCLUSION
It is therefore determined that the instant application not only represents an abstract idea identified as such based on criteria defined by the Courts and on USPTO examination guidelines, but also lacks the capability to bring about "Improvements to another technology or technical field" (Alice), bring about "Improvements to the functioning of the computer itself" (Alice), "Apply the judicial exception with, or by use of, a particular machine" (Bilski), "Effect a transformation or reduction of a particular article to a different state or thing" (Diehr), "Add a specific limitation other than what is well-understood, routine and conventional in the field" (Mayo), "Add unconventional steps that confine the claim to a particular useful application" (Mayo), or contain "Other meaningful limitations beyond generally linking the use of the judicial exception to a particular technological environment" (Alice), transformed a traditionally subjective process performed by humans into a mathematically automated process executed on computers (McRO), or limitations directed to improvements in computer related technology, including claims directed to software (Enfish).
The dependent claims, when considered individually and as a whole, likewise do not provide “significantly more” than the abstract idea for similar reasons as the independent claim. For example claim 2 disclose “wherein the first region is a region within a range of a predetermined distance from an imaging device that captures the image, and the second region is a region set at a position further than the predetermined distance from the imaging device” data description , which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use. It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea; claim 3 disclose gesture recognition system according to claim 1, wherein the first information is information for recognizing a gesture that does not include a motion of an arm, include a motion of the hand or fingers, and is achieved by a motion of the hand or the fingers. data description , which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use. It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea; claim 4 disclose wherein the second information is information for recognizing a gesture that includes a motion of an arm. data description , which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use. It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea. claim 5 disclose gesture recognition system according to claim 5, wherein the first region is a region in which it is not possible or difficult to recognize the motion of the arm of the user from the image capturing the user who is present in the first region through execution of the instructions by the one or more processors, data description and using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use. It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea. claim 6 disclose recognize a gesture of the user on the basis of the image, the first information, and the second information in a case in which the user is present in a third region which is located across the first region and a second region that is outside the first region and is adjacent to the first region or a third region located between the first region and a second region that is located further than the first region when the image is captured (mental process) and using processor (using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 7 disclose wherein the one or more processors execute the instructions (using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) to recognize a gesture of the user by placing higher priority on a result of recognition based on the image and the first information than on a result of recognition based on the image and the second information (mental process) that does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea. claim 9 disclose a second storage device storing reference information in which a gesture of the user and an operation of the mobile object are associated (mental process); and at least one processor configured to control the mobile object on the basis of the operation of the mobile object associated with the gesture of the user with reference to the reference information instructions (mental process of controlling based on data analysis and using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use). Does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea. claim 10 disclose a first imager configured to image surroundings of the mobile object; and a second imager configured to image a user who remotely operates the mobile object, wherein the one or more processors execute the instructions (using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) to attempt processing for recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager and employ, with higher priority, a result of the recognition based on the second image than a result of the recognition on the basis of the first image (mental process), and cause the mobile object to be controlled on the basis of a surrounding situation obtained from the image captured by the first imager and the operation associated with the gesture recognized by the recognizer (do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2) in addition it could fall also as an (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)). Claim 10 does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea, claim 12 disclose wherein the one or more processors execute the instructions (using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) to track a user as a target on the basis of a captured image (mental process), recognize a gesture of the user who is being tracked, and not perform processing for recognizing gestures of persons who are not being tracked (mental process), and control the mobile object on the basis of the gesture of the user who is being tracked (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)). Claim 12 does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea. claim 15 disclose wherein the one or more processors execute the instructions (using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) to: receive registration of the user; and acquire an image capturing the user (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)). Claim 15 does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea. claim 16 disclose a storage device configured to store instructions; and one or more processors, wherein the one or more processors execute the instructions stored in the storage device (using processor as apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) to acquire an image capturing a user (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)), recognize a region where the user is present when the image is captured, and in a case in which the user is present in a first region when the image is captured, recognize a gesture of the user on the basis of the image and first information for recognizing the gesture of the user (mental process), and in a case in which the user is present in a second region when the image is captured, recognize a gesture of the user on the basis of the image and second information for recognizing the gesture of the user (mental process), wherein gesture recognition system further comprising: a first imager configured to image surroundings of the mobile object; a second imager configured to image a user who remotely operates the mobile object (data description , which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use) and, a storage device configured to store reference information in which a gesture of the user and an operation of the mobile object are associated (apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use), control the mobile object on the basis of the operation of the mobile object associated with the gesture of the user with reference to the reference information (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)), wherein the first imager is different with the second imager(apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use), the second camera is capable of controlling the imaging direction using a machine mechanism, captures an image around the user as a tracking target at the center (apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use), attempt processing for recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager and employ, with higher priority, a result of the recognition based on the second image than a result of the recognition on the basis of the first image,(mental process) and cause the mobile object to be controlled on the basis of a surrounding situation obtained from the image captured by the first imager and the operation associated with the gesture recognized (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)). Claim 16 does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea. claim 17 disclose gesture recognition system comprising: a storage device configured to store instructions; and one or more processors, wherein the one or more processors execute the instructions stored in the storage device to acquire an image capturing a user (apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use), recognize a region where the user is present when the image is captured, and in a case in which the user is present in a first region when the image is captured, recognize a gesture of the user on the basis of the image and first information for recognizing the gesture of the user, and in a case in which the user is present in a second region when the image is captured, recognize a gesture of the user on the basis of the image and second information for recognizing the gesture of the user (mental process), wherein gesture recognition system further comprising: a first imager configured to image surroundings of the mobile object; a second imager configured to image a user who remotely operates the mobile object, and, a storage device configured to store reference information in which a gesture of the user and an operation of the mobile object are associated (apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use), control the mobile object on the basis of the operation of the mobile object associated with the gesture of the user with reference to the reference information, wherein the first imager is different with the second imager (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)), the second camera is capable of controlling the imaging direction using a machine mechanism, captures an image around the user as a tracking target at the center (apply it, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use), in a case that recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager when the user is in the first region, control the mobile object on the basis of the gesture of the user on the basis of the second image captured (mental process), in a case that recognizing a gesture of the user on the basis of a first image captured by the first imager and not recognizing the gesture of the user on the basis of a second image captured by the second imager when the user is in the first region (mental process), control the mobile object on the basis of the gesture of the user on the basis of the first image captured (insignificant extra-solution activity, MPEP 2106.05(g)) for transmitting commands that is Well-Understood-Routine, Conventional activity sending, receiving, displaying and processing data are common and basic functions in computer technology, MPEP 2106.05(d)(II)(i)). Claim 16 does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea.
Therefore the dependent claims which impose additional limitations also fail to claim patent-eligible subject matter because the limitations cannot be considered statutory. In reference to the dependent claim have also been reviewed with the same analysis as independent claim 1. The dependent claim(s) have been examined individually and in combination with the preceding claims, however they do not cure the deficiencies of claim 1; where all claims are directed to the same abstract idea, "addressing each claim of the asserted patents [is] unnecessary." Content Extraction &. Transmission LLC v, Wells Fargo Bank, Natl Ass'n, 776 F.3d 1343, 1348 (Fed. Cir. 2014). If applicant believes the dependent claims are directed towards patent eligible subject matter, they are invited to point out the specific limitations in the claim that are directed towards patent eligible subject matter. Claims 13 and 14 for the other statutory class are similarly analyzed. Examiner notes that claim 14 further disclose “wherein the trajectory is further generated by connecting a first arc and a second arc of the group of arcs, and the first arc and the second arc are trajectories that satisfy a group of conditions comprising: a forward direction of the mobile object is a X direction, a X line is an imaginary line extending in the X direction, a lateral direction of the mobile object is a Y direction, a Y line is imaginary line perpendicularly intersecting the X direction and extending in the Y direction, a first end point of the first arc is Zm1, a tangential line representing Zm1 is a first tangential line, a second end point of the second arc is Zm2, a tangential line representing Zm2 is a second tangential line, a first angle formed by a first line perpendicularly intersecting the first tangential line and extending in the Y direction and the Y line is represented as ϴ1,a second angle formed by a second line perpendicularly intersecting the second tangential line and extending in the Y direction and the first line is represented as ϴ2, and a third angle formed by the second line and the X line is ϴ1 +ϴ2.” These limitations is directed to the description of data, which is directed to generally linking the use of a judicial exception to a particular technological environment or field of use. It does not integrate the abstract idea into a practical application and did not add significantly more to the abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5, 9, and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. [US 2019/0155313 A1, hereinafter Tang] in view of Goulding [US 2011/0231050 A1] in view of Anezaki [US 2010/0222925 A1] in view of Savarit et al. [US 2018/0012502 A1, hereinafter Savarit]
With regard to Claim 1,
Tang teach a gesture recognition system comprising:
a storage device configured to store instructions (202); and
one or more processors, wherein the one or more processors execute the instructions stored in the storage device to (Fig. 2, 201, ¶7, ¶30)
acquire an image capturing a user, wherein the image is pixel data representative of camera coordinate system data (¶7, “programming a computer-readable medium with instructions that, when executed, can generate an image corresponding to an operator”, ¶5, “image-collection component can be a color-sensing camera that can be used to collect color images (e.g., those having red, green, and blue (RGB) pixels.)”),
recognize a region where the user is present when the image is captured (¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator”), and
in a case in which the user is present in a first region when the image is captured, recognize a gesture of the user on the basis of the image and first information for recognizing the gesture of the user (¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator. The instructions can identify a portion of the image corresponding to the operator and retrieve a second set of distance information from the first set of distance information, based at least in part on the identified portion of the image corresponding to the operator”, ¶22, “One factor to consider when determining a type of the preferred gesture is the distance between the moveable device and the operator. For example, when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode)”), wherein the gesture of the user generated by the user initiates a movement of a mobile object (¶6. “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction”); and
in a case in which the user is present in a second region when the image is captured, recognize a gesture of the user on the basis of the image and second information for recognizing the gesture of the user (¶7, ¶22, “when the distance between the moveable device and the operator is greater than the pre-determined range, the system can select an “arm gesture” as the preferred gesture (i.e., in a long distance mode”);
in response to recognizing the gesture, determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point (¶6. “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction”), wherein the trajectory between the first trajectory point and the second trajectory point is determined (¶6, “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction”), and initiating the movement of the mobile object in accordance with the trajectory determined based on [identified trajectory] (¶6. “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction”).
Tang does not explicitly teach wherein
Goulding wherein the gesture of the user and a defined sound generated by the user initiates [command], and recognizing the gesture and the defined sound (¶465, “High-level control is input via external commands through the network interface 95, through body language of the at least one rider or passenger 101 as measured by the force and torque sensors 81 and IMU 82, an external control device, such as for example a radio control unit, voice commands, or visual commands or gestures, or any combination of such devices and sensing”, ¶483, ¶487, “ridden by a human operator that communicates with the robot via body motions, verbal commands, and an interface (e.g., handle bars and hand-grip controls)”).
Tang and Goulding are analogous art to the claimed invention because they are from a similar field of endeavor of controlling mobile devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang resulting in resolutions as disclosed by Goulding with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang as described above to provide the ability to Mimics how people naturally communicate, making technology feel more responsive and less demanding, empowers users with mobility or visual impairments to control devices independently, offering alternatives to traditional keyboards/mice, improve productivity by allowing users to perform complex tasks more quickly, and provide contextual understanding as gestures (like pointing) add spatial context to voice commands, clarifying intent. This is simply use of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Tang-Goulding does not explicitly teach conversion of coordinate representations of the first trajectory and the second trajectory.
Anezaki teach image is pixel data representative of camera coordinate system data (¶161, “in the case of an omnidirectional camera image of 256*256 pixel size, the corresponding points 301 are arranged at intervals of about 16 pixels on the outer circle's perimeter of the omnidirectional camera image”);
determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point (¶128, “moving path outputted from the moving path generation unit 37, and operate-controlling the mobile robot 1 so that the mobile robot 1 can travel accurately to the target point without deviating the moving path which is the normal route”, ¶197, “travel distance data detected by the travel distance detection unit 20 and travel direction data detected by the directional angle detection unit 30 are inputted to the control unit 50 at predetermined time intervals, and the current position of the robot 1 is calculated by the control unit 50. Displacement information … calculated by the displacement information calculation unit 40 is inputted into the control unit 50, and according to the displacement information result, the control unit 50 controls the drive unit 10 to control the moving path of the robot 1”) wherein the trajectory between the first trajectory point and the second trajectory point is determined based on a conversion of coordinate representations of the first trajectory point and the second trajectory point into the camera coordinate system data representative of the first trajectory point and the second trajectory point (Fig. 15-18, Fig. 23-26, ¶109, “characteristics of an input image when using a PAL-type lens or a fish-eye lens in one example of an omnidirectional image input unit”, ¶206, “omnidirectional camera processing unit 412 having such a configuration, matching is performed … positional posture shift detection (detection of rotational angle-shifted amount obtained by the rotational angle-shifted amount conversion unit and displacement amount obtained by the displacement amount conversion unit) is performed, thus the robot's position is recognized, and the drive unit 10 is drive-controlled by the control unit 50 so as to correct the rotational angle-shifted amount and the displacement amount to be in an allowable area respectively, whereby the robot control apparatus controls the robot 1 to move autonomously”, ¶209), and
initiating the movement the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory and the second trajectory (Fig. 23-24, ¶109, “characteristics of an input image when using a PAL-type lens or a fish-eye lens in one example of an omnidirectional image input unit”, ¶206, “omnidirectional camera processing unit 412 having such a configuration, matching is performed … positional posture shift detection (detection of rotational angle-shifted amount obtained by the rotational angle-shifted amount conversion unit and displacement amount obtained by the displacement amount conversion unit) is performed, thus the robot's position is recognized, and the drive unit 10 is drive-controlled by the control unit 50 so as to correct the rotational angle-shifted amount and the displacement amount to be in an allowable area respectively, whereby the robot control apparatus controls the robot 1 to move autonomously”, ¶78).
Tang-Goulding and Anezaki are analogous art to the claimed invention because they are from a similar field of endeavor of controlling movable devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding resulting in resolutions as disclosed by Anezaki with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding as described above to provide the ability to create a clear movement path by removing surrounding distortion of the omnidirectional image input (Anezaki, ¶210).
Tang-Goulding-Anezaki does not explicitly teach trajectory is further determined by combining an arc of a group of arcs representative of the trajectory, wherein the arc of the group of arcs has a different curvature radii from other arcs comprising the group of arcs, wherein the arc of the group arcs is associated with a predicted time period of a sequence of predicted time periods, and wherein the predicted time period indicates a defined position on the trajectory along which the mobile object moves, and initiating the movement the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory point and the second trajectory point; and based on the trajectory determined based on the conversion of coordinate representations, causing the mobile object to move along the trajectory from the first trajectory point to the second trajectory point.
Savarit teach determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point (¶15, “flight plan comprises an ordered series of segments … trajectory arising from the flight plan PV is constructed gradually from segment to segment on the basis of the directives contained in each segment (inter-waypoint geometry defined by these segments)”, ¶17, “Each segment thus generates a portion of trajectory or elementary trajectory … a straight section, an arc, typically a circular arc, or combinations of straight section and arcs”, ¶21), wherein the trajectory between the first trajectory point and the second trajectory point is determined based on a conversion of coordinate representations of the first trajectory point and the second trajectory point into the camera coordinate system data representative of the first trajectory point and the second trajectory point (¶15, “flight plan comprises an ordered series of segments … trajectory arising from the flight plan PV is constructed gradually from segment to segment on the basis of the directives contained in each segment (inter-waypoint geometry defined by these segments)”, ¶17, “Each segment thus generates a portion of trajectory or elementary trajectory … a straight section, an arc, typically a circular arc, or combinations of straight section and arcs”, ¶102, “module PRED a vertical trajectory as a function typically of a curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement, taking account of the speed of the aircraft)”, , ¶132, ¶¶104-105, ¶109, system allow and use conversion of coordinate representations of the first trajectory point and the second trajectory point between different coordinates as geometry, arc, angular curvilinear abscissa, etc. ), wherein the trajectory is further determined by combining an arc of a group of arcs representative of the trajectory, wherein the arc of the group of arcs has a different curvature radii from other arcs comprising the group of arcs (¶92, “making it possible to construct a “multi-radii” transition”, ¶109, “arc of the improved transition, that is to say its curved part, is composed of several circular arcs of radius R(i) which follow one another in a continuous and ordered manner for i=1 to N”, ¶110, “arc consisting of 3 circular arcs of increasing respective radii R1, R2 and R3”, ¶112, ¶115, “Timp constructed comprises a first arc consisting of two circular arcs respectively of radii R1, R2 … and of a second arc consisting of two circular arcs, respectively of radii R3 and R4”, Fig. 7-8), wherein the arc of the group arcs is associated with a predicted time period of a sequence of predicted time periods (¶19, “On the basis of the complete calculation of the trajectory, the FMS determines “predictions” … that is to say for various values of the curvilinear abscissa x of the trajectory”, ¶102, “curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement, taking account of the speed of the aircraft). The predictions … are the values taken by these parameters at certain curvilinear abscissae of the trajectory”, ¶104, “for each subdivision Sub(i), an associated turning radius R(i) is determined, calculated on the basis of a representative value PA(i), PB(i), PC(i), PD(i) of each parameter for the subdivision i, extrapolated on the basis of the predicted values, determined in step 3”, each subdivision (arc segment) use prediction based on x that is distance or time elapsed, therefore each arc is associated with a predicted (elapsed) time window (sequence of predicted time periods in the subdivisions). This produce predicted sequence of intervals (predicted time periods), and this identify position via X), and wherein the predicted time period indicates a defined position on the trajectory along which the mobile object moves (¶102, “curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement …), ¶141, “Typically x is the curvilinear abscissa of the transition”), and
Initiating the movement of the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory point and the second trajectory point (¶20, “FMS can, on instruction from the pilot, slave the aircraft automatically to the calculated trajectory”, ¶11, “Guidance (GU ID) 107, for guiding the aircraft in the lateral and vertical planes on its three-dimensional trajectory”); and
based on the trajectory determined based on the conversion of coordinate representations, causing the mobile object to move along the trajectory from the first trajectory point to the second trajectory point (Fig. 2, ¶31, “aircraft is slaved to the calculated trajectory which is frozen”, ¶20, “FMS can, on instruction from the pilot, slave the aircraft automatically to the calculated trajectory”, ¶35, “aircraft flying at nominal roll angle φN, the trajectory radius actually flown by the aircraft”, ¶117, “An improved trajectory Traj-imp incorporating the improved transition Timp is determined in step 7), and the improved trajectory Traj-imp is displayed to a pilot of the aircraft”, mobile object moves between trajectory points).
Tang-Goulding-Anezaki and Savarit are analogous art to the claimed invention because they are from a similar field of endeavor of controlling movable devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-Anezaki resulting in resolutions as disclosed by Savarit with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-Anezaki as described above for calculating a lateral trajectory some of whose transitions are improved (Savarit, ¶2).
With regard to Claim 2,
Tang-Goulding-Anezaki-Savarit teach the gesture recognition system according to claim 1, wherein the first region is a region within a range of a predetermined distance from an imaging device that captures the image (Tang, ¶22, “One factor to consider when determining a type of the preferred gesture is the distance between the moveable device and the operator. For example, when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode)”), and the second region is a region set at a position further than the predetermined distance from the imaging device (Tang, ¶22, “when the distance between the moveable device and the operator is greater than the pre-determined range, the system can select an “arm gesture” as the preferred gesture (i.e., in a long distance mode”). The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 3,
Tang-Goulding-Anezaki-Savarit teach the gesture recognition system according to claim 1, wherein the first information is information for recognizing a gesture that does not include a motion of an arm, include a motion of the hand or fingers, and is achieved by a motion of the hand or the fingers (Tang, ¶22, “One factor to consider when determining a type of the preferred gesture is the distance between the moveable device and the operator. For example, when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode)”). The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 4,
Tang-Goulding-Anezaki-Savarit teach the gesture recognition system according to claim 1, wherein the second information is information for recognizing a gesture that includes a motion of an arm (Tang, ¶22, “when the distance between the moveable device and the operator is greater than the pre-determined range, the system can select an “arm gesture” as the preferred gesture (i.e., in a long distance mode”). The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 5,
Tang-Goulding-Anezaki-Savarit teach the gesture recognition system according to claim 4, wherein the first region is a region in which it is not possible or difficult to recognize the motion of the arm of the user from the image capturing the user who is present in the first region through execution of the instructions by the one or more processors (Tang, ¶22, “when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode). This is because the system has a better chance of collecting images of the operator's whole hand than collecting images of the operator's whole arm”). The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 9,
Tang-Goulding-Anezaki-Savarit teach the gesture recognition system according to claim 1, further comprising a mobile object, comprising:
a second storage device storing reference information in which a gesture of the user and an operation of the mobile object are associated (Tang, Fig. 2, ¶21, “associated system can compare the collected images and measured distance information with reference information or a pre-determined dataset stored in a pre-existing database (e.g., can be a remote database or data storage in the system)”, ¶37, “collected images and the measured distance information can be related or associated such that the system 200 can utilize both to identify (and, optionally, verify) a gesture by the operator”, ¶¶38-39. ¶41, “verification component 209 can analyze the stored images to see if the identified gesture in these images is consistent from one frame to another. If so, the system 200 can move forward with the instruction generation component 211 forming a corresponding instruction based on the identified gesture”, “verification component 209 can verify the identified gesture by comparing it with reference information (e.g., a reference set of hand gesture depth information from multiple operators) located in a (remote or local) database”); and
at least one processor configured to control the mobile object on the basis of the operation of the mobile object associated with the gesture of the user with reference to the reference information (Tang, Fig. 2, ¶30, ¶37, “collected images and the measured distance information can be related or associated such that the system 200 can utilize both to identify (and, optionally, verify) a gesture by the operator”, ¶¶38-39. ¶41, “verification component 209 can analyze the stored images to see if the identified gesture in these images is consistent from one frame to another. If so, the system 200 can move forward with the instruction generation component 211 forming a corresponding instruction based on the identified gesture”). The same motivation to combine for claim 1 equally applies for current claim.
With regard to Claim 13,
Claim 13 is similar in scope to claim 1; therefore they are rejected and similar rationale.
With regard to Claim 14,
Tang teach a gesture recognition system comprising:
a storage device configured to store instructions (202); and
one or more processors, wherein the one or more processors execute the instructions stored in the storage device to (Fig. 2, 201, ¶7, ¶30)
acquire an image capturing a user, wherein the image is pixel data representative of camera coordinate system data (¶7, “programming a computer-readable medium with instructions that, when executed, can generate an image corresponding to an operator”, ¶5, “image-collection component can be a color-sensing camera that can be used to collect color images (e.g., those having red, green, and blue (RGB) pixels.)”),
recognize a region where the user is present when the image is captured (¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator”), and
in a case in which the user is present in a first region when the image is captured, recognize a gesture of the user on the basis of the image and first information for recognizing the gesture of the user (¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator. The instructions can identify a portion of the image corresponding to the operator and retrieve a second set of distance information from the first set of distance information, based at least in part on the identified portion of the image corresponding to the operator”, ¶22, “One factor to consider when determining a type of the preferred gesture is the distance between the moveable device and the operator. For example, when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode)”); and
in a case in which the user is present in a second region when the image is captured, recognize the gesture of the user on the basis of the image and second information for recognizing the gesture of the user (¶7, ¶22, “when the distance between the moveable device and the operator is greater than the pre-determined range, the system can select an “arm gesture” as the preferred gesture (i.e., in a long distance mode”);
in response to recognizing the gesture, determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point (¶6. “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction”), wherein the trajectory between the first trajectory point and the second trajectory point is determined (¶6, “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction.
Tang does not explicitly teach a forward direction of the mobile object is a X direction, a X line is an imaginary line extending in the X direction, a lateral direction of the mobile object is a Y direction, a Y line is imaginary line perpendicularly intersecting the X direction and extending in the Y direction.
Goulding wherein a forward direction of the mobile object is a X direction, a X line is an imaginary line extending in the X direction (¶436, “FIG. 4 illustrates the side skeletal view … wherein the x and z axes with the side or y axis into and out of the page. By convention, the left side is the forward direction”, ¶324, “propels the body in the x, y, and z axes”, ¶433, ” positioned in six dimensions (X-Y-Z and roll-pitch-yaw axis, respectively)”, ¶435, “center of gravity 58 and its projection to the ground 55, called the center of pressure 59”), a lateral direction of the mobile object is a Y direction (¶436, “FIG. 4 illustrates the side skeletal view … wherein the x and z axes with the side or y axis into and out of the page” ¶324, “propels the body in the x, y, and z axes”, ¶433, ” positioned in six dimensions (X-Y-Z and roll-pitch-yaw axis, respectively)”), a Y line is imaginary line perpendicularly intersecting the X direction and extending in the Y direction (¶436, “FIG. 4 illustrates the side skeletal view … wherein the x and z axes with the side or y axis into and out of the page” ¶324, “propels the body in the x, y, and z axes”, ¶433, ” positioned in six dimensions (X-Y-Z and roll-pitch-yaw axis, respectively)”).
Tang and Goulding are analogous art to the claimed invention because they are from a similar field of endeavor of controlling mobile devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang resulting in resolutions as disclosed by Goulding with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang as described above to provide high precision in straight-line moves, cost-effectiveness, easy customization, and simpler programming/maintenance, making them ideal for tasks needing rigidity and accuracy. This is simply use of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Tang-Goulding does not explicitly teach conversion of coordinate representations of the first trajectory and the second trajectory.
Anezaki teach image is pixel data representative of camera coordinate system data (¶161, “in the case of an omnidirectional camera image of 256*256 pixel size, the corresponding points 301 are arranged at intervals of about 16 pixels on the outer circle's perimeter of the omnidirectional camera image”);
determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point (¶128, “moving path outputted from the moving path generation unit 37, and operate-controlling the mobile robot 1 so that the mobile robot 1 can travel accurately to the target point without deviating the moving path which is the normal route”, ¶197, “travel distance data detected by the travel distance detection unit 20 and travel direction data detected by the directional angle detection unit 30 are inputted to the control unit 50 at predetermined time intervals, and the current position of the robot 1 is calculated by the control unit 50. Displacement information … calculated by the displacement information calculation unit 40 is inputted into the control unit 50, and according to the displacement information result, the control unit 50 controls the drive unit 10 to control the moving path of the robot 1”) wherein the trajectory between the first trajectory point and the second trajectory point is determined based on a conversion of coordinate representations of the first trajectory point and the second trajectory point into the camera coordinate system data representative of the first trajectory point and the second trajectory point (Fig. 15-18, Fig. 23-26, ¶109, “characteristics of an input image when using a PAL-type lens or a fish-eye lens in one example of an omnidirectional image input unit”, ¶206, “omnidirectional camera processing unit 412 having such a configuration, matching is performed … positional posture shift detection (detection of rotational angle-shifted amount obtained by the rotational angle-shifted amount conversion unit and displacement amount obtained by the displacement amount conversion unit) is performed, thus the robot's position is recognized, and the drive unit 10 is drive-controlled by the control unit 50 so as to correct the rotational angle-shifted amount and the displacement amount to be in an allowable area respectively, whereby the robot control apparatus controls the robot 1 to move autonomously”, ¶209), and
initiating the movement the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory and the second trajectory (Fig. 23-24, ¶109, “characteristics of an input image when using a PAL-type lens or a fish-eye lens in one example of an omnidirectional image input unit”, ¶206, “omnidirectional camera processing unit 412 having such a configuration, matching is performed … positional posture shift detection (detection of rotational angle-shifted amount obtained by the rotational angle-shifted amount conversion unit and displacement amount obtained by the displacement amount conversion unit) is performed, thus the robot's position is recognized, and the drive unit 10 is drive-controlled by the control unit 50 so as to correct the rotational angle-shifted amount and the displacement amount to be in an allowable area respectively, whereby the robot control apparatus controls the robot 1 to move autonomously”, ¶78).
Tang-Goulding and Anezaki are analogous art to the claimed invention because they are from a similar field of endeavor of controlling movable devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding resulting in resolutions as disclosed by Anezaki with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding as described above to provide the ability to create a clear movement path by removing surrounding distortion of the omnidirectional image input (Anezaki, ¶210).
Tang-Goulding-Anezaki does not explicitly teach trajectory is further determined by combining an arc of a group of arcs representative of the trajectory, wherein the arc of the group of arcs has a different curvature radii from other arcs comprising the group of arcs, wherein the arc of the group arcs is associated with a predicted time period of a sequence of predicted time periods, and wherein the predicted time period indicates a defined position on the trajectory along which the mobile object moves, and initiating the movement the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory point and the second trajectory point; and based on the trajectory determined based on the conversion of coordinate representations, causing the mobile object to move along the trajectory from the first trajectory point to the second trajectory point, a first end point of the first arc is Zm1, a tangential line representing Zm1 is a first tangential line, a second end point of the second arc is Zm2, a tangential line representing Zm2 is a second tangential line, a first angle formed by a first line perpendicularly intersecting the first tangential line and extending in the Y direction and the Y line is represented as ϴ1,a second angle formed by a second line perpendicularly intersecting the second tangential line and extending in the Y direction and the first line is represented as ϴ2, and a third angle formed by the second line and the X line is ϴ1 +ϴ2.
Savarit teach determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point (¶15, “flight plan comprises an ordered series of segments … trajectory arising from the flight plan PV is constructed gradually from segment to segment on the basis of the directives contained in each segment (inter-waypoint geometry defined by these segments)”, ¶17, “Each segment thus generates a portion of trajectory or elementary trajectory … a straight section, an arc, typically a circular arc, or combinations of straight section and arcs”, ¶21), wherein the trajectory between the first trajectory point and the second trajectory point is determined based on a conversion of coordinate representations of the first trajectory point and the second trajectory point into the camera coordinate system data representative of the first trajectory point and the second trajectory point (¶15, “flight plan comprises an ordered series of segments … trajectory arising from the flight plan PV is constructed gradually from segment to segment on the basis of the directives contained in each segment (inter-waypoint geometry defined by these segments)”, ¶17, “Each segment thus generates a portion of trajectory or elementary trajectory … a straight section, an arc, typically a circular arc, or combinations of straight section and arcs”, ¶102, “module PRED a vertical trajectory as a function typically of a curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement, taking account of the speed of the aircraft)”, , ¶132, ¶¶104-105, ¶109, system allow and use conversion of coordinate representations of the first trajectory point and the second trajectory point between different coordinates as geometry, arc, angular curvilinear abscissa, etc. ), wherein the trajectory is further determined by combining an arc of a group of arcs representative of the trajectory, wherein the arc of the group of arcs has a different curvature radii from other arcs comprising the group of arcs (¶92, “making it possible to construct a “multi-radii” transition”, ¶109, “arc of the improved transition, that is to say its curved part, is composed of several circular arcs of radius R(i) which follow one another in a continuous and ordered manner for i=1 to N”, ¶110, “arc consisting of 3 circular arcs of increasing respective radii R1, R2 and R3”, ¶112, ¶115, “Timp constructed comprises a first arc consisting of two circular arcs respectively of radii R1, R2 … and of a second arc consisting of two circular arcs, respectively of radii R3 and R4”, Fig. 7-8), wherein the arc of the group arcs is associated with a predicted time period of a sequence of predicted time periods (¶19, “On the basis of the complete calculation of the trajectory, the FMS determines “predictions” … that is to say for various values of the curvilinear abscissa x of the trajectory”, ¶102, “curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement, taking account of the speed of the aircraft). The predictions … are the values taken by these parameters at certain curvilinear abscissae of the trajectory”, ¶104, “for each subdivision Sub(i), an associated turning radius R(i) is determined, calculated on the basis of a representative value PA(i), PB(i), PC(i), PD(i) of each parameter for the subdivision i, extrapolated on the basis of the predicted values, determined in step 3”, each subdivision (arc segment) use prediction based on x that is distance or time elapsed, therefore each arc is associated with a predicted (elapsed) time window (sequence of predicted time periods in the subdivisions). This produce predicted sequence of intervals (predicted time periods), and this identify position via X), and wherein the predicted time period indicates a defined position on the trajectory along which the mobile object moves (¶102, “curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement …), ¶141, “Typically x is the curvilinear abscissa of the transition”), and
Initiating the movement of the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory point and the second trajectory point (¶20, “FMS can, on instruction from the pilot, slave the aircraft automatically to the calculated trajectory”, ¶11, “Guidance (GU ID) 107, for guiding the aircraft in the lateral and vertical planes on its three-dimensional trajectory”); and
based on the trajectory determined based on the conversion of coordinate representations, causing the mobile object to move along the trajectory from the first trajectory point to the second trajectory point (Fig. 2, ¶31, “aircraft is slaved to the calculated trajectory which is frozen”, ¶20, “FMS can, on instruction from the pilot, slave the aircraft automatically to the calculated trajectory”, ¶35, “aircraft flying at nominal roll angle φN, the trajectory radius actually flown by the aircraft”, ¶117, “An improved trajectory Traj-imp incorporating the improved transition Timp is determined in step 7), and the improved trajectory Traj-imp is displayed to a pilot of the aircraft”, mobile object moves between trajectory points),
wherein the trajectory is further generated by connecting a first arc and a second arc of the group of arcs, and the first arc and the second arc are trajectories that satisfy a group of conditions (¶109, “Next, in a step 6), an improved transition Timp is determined on the basis of the ordered subdivisions Sub(i) and of the successive associated turning radii R(i). The arc of the improved transition, that is to say its curved part, is composed of several circular arcs of radius R(i) which follow one another in a continuous and ordered manner for i=1 to N, N being the number of subdivisions”, ¶115, “The improved transition Timp constructed comprises a first arc consisting of two circular arcs respectively of radii R1, R2 … and of a second arc consisting of two circular arcs, respectively of radii R3 and R4”) comprising:
a forward direction of the mobile object is a X direction, a X line is an imaginary line extending in the X direction (¶43, “… a trajectory being determined on the basis of a flight plan … the trajectory being constructed gradually on the basis of the directives contained in each segment, a trajectory portion making it possible to link the elementary trajectories corresponding to two nonaligned consecutive segments being termed a transition”, ¶102, “the module PRED of the FMS. The module TRAJ calculates a lateral trajectory, and the module PRED a vertical trajectory as a function typically of a curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory”), a lateral direction of the mobile object is a Y direction, a Y line is imaginary line perpendicularly intersecting the X direction and extending in the Y direction (¶2, “ relates to a method for calculating a lateral trajectory some of whose transitions (changes of direction of the aircraft) are improved”, ¶110, “construction for an initial transition Tini between a point WPA and a heading HD. The initial transition Tini is a … circular arc of radius R0 and of angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, ¶114, “the two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, system use lateral trajectory and disclose angular sectors (i.e. angle in 2D) that require perpendicular axes),
a first end point of the first arc is Zm1, a tangential line representing Zm1 is a first tangential line (¶115, “The improved transition Timp constructed comprises a first arc consisting of two circular arcs respectively of radii R1, R2 (in replacement for the first circular arc of radius R0), of a straight line portion and of a second arc”, ¶117, “An improved trajectory Traj-imp incorporating the improved transition Timp is determined in step 7), and the improved trajectory Traj-imp is displayed to a pilot of the aircraft in step 8). The trajectory is constructed gradually, so as to be continuous and differentiable”, Being differentiable at end point of first arc means a function has a well-defined, non-vertical tangent line (its best linear approximation) at that specific at end point of the first arc),
a second end point of the second arc is Zm2, a tangential line representing Zm2 is a second tangential line (¶115, “and of a second arc consisting of two circular arcs, respectively of radii R3 and R4 (in replacement for the second circular arc of radius R0).”, ¶117, “… The trajectory is constructed gradually, so as to be continuous and differentiable”, Being differentiable at end point of second arc means a function has a well-defined, non-vertical tangent line (its best linear approximation) at that specific at end point of second arc),
a first angle formed by a first line perpendicularly intersecting the first tangential line and extending in the Y direction and the Y line is represented as ϴ1 (¶114, “two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, ¶110, “construction for an initial transition Tini between a point WPA and a heading HD. The initial transition Tini is a … circular arc of radius R0 and of angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, ¶113, “Tini consists of a first circular arc of radius R0 (turn to the right) of angular sector equal to 135°, of a straight line portion and of a second circular arc of radius R0 (turn to the left) of angular sector equal to 45°”, direction of airplane at the beginning of the turning arc rotates as the aerplane enters the first subdivision of the transition arc (ϴ1). As the airplane turns, the tangent direction at the first end point rotate relative to Y line and the angular change occurring within this first subdivision corresponding to ϴ1, this is clarified by explicitly disclosing that the transition arc subdivided into a smaller angular sectors where the first subdivision of the subdivisions is ϴ1),
a second angle formed by a second line perpendicularly intersecting the second tangential line and extending in the Y direction and the first line is represented as ϴ2 (¶114, “two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, ¶110, “construction for an initial transition Tini between a point WPA and a heading HD. The initial transition Tini is a … circular arc of radius R0 and of angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, second subdivision produces the second incremental turn angle),
and a third angle formed by the second line and the X line is ϴ1 +ϴ2 (¶132, “ the criterion consists in determining a number of subdivisions N, each subdivision corresponding to an angular sector of angle equal α, the angle α corresponding to the angular sector of the initial transition θ0 divided by the number of subdivisions N”, ¶110, “angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, ¶114, “the two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, transition turn not flown in a single big angle but in pieces, because the FMS divides the arc into sub-arcs, each with its own turning radius. So the first sub-arc produce first portion of the turn ϴ1, second sub-arc produce second portion of the turn ϴ2, third angle represent the entire turn from start to finish ϴ1+ϴ2).
Tang-Goulding-Anezaki and Savarit are analogous art to the claimed invention because they are from a similar field of endeavor of controlling movable devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-Anezaki resulting in resolutions as disclosed by Savarit with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-Anezaki as described above for calculating a lateral trajectory some of whose transitions are improved (Savarit, ¶2).
With regard to Claim 15,
Tang-Goulding-Anezaki-Savarit teach the gesture recognition system according to claim 1, wherein the one or more processors execute the instructions to: receive registration of the user (¶23, “operator may need to form a predetermined gesture (or a combination of gestures) to “unlock” the associated system … associated system can verify the identity of the user based on a security device (e.g., a facial recognition device, a finger print reader, an audio recognition device, etc.) associated with the system”, ¶24, “operator can be selected or identified via the additional controller (e.g., via a user interface provided by the app installed in the smartphone) … operator can be verified by the additional controller”); and acquire an image capturing the user (¶5, “image-collection component can be a color-sensing camera that can be used to collect color images (e.g., those having red, green, and blue (RGB) pixels.)”). The same motivation to combine for claim 1 equally applies for current claim.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. [US 2019/0155313 A1, hereinafter Tang] in view of Goulding [US 2011/0231050 A1] in view of Anezaki [US 2010/0222925 A1] in view of Savarit et al. [US 2018/0012502 A1, hereinafter Savarit] in view of Takayanagi et al. [US 2018/0046254 A1, hereinafter Takayanagi]
With regard to Claim 6,
Tang-Goulding-Anezaki-Savarit teach the gesture recognition system according to claim 1,
wherein the one or more processors execute the instructions to
recognize a gesture of the user on the basis of the image, the first information, and the second information (Tang, ¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator. The instructions can identify a portion of the image corresponding to the operator and retrieve a second set of distance information from the first set of distance information, based at least in part on the identified portion of the image corresponding to the operator”, ¶22, “One factor to consider when determining a type of the preferred gesture is the distance between the moveable device and the operator”). The same motivation to combine for claim 1 equally applies for current claim.
Tang-Goulding-Anezaki-Savarit does not explicitly teach the in a case in which the user is present in a third region which is located across the first region and a second region that is outside the first region and is adjacent to the first region or a third region located between the first region and a second region that is located further than the first region when the image is captured.
Takayanagi teach recognize a gesture of the user on the basis of the image, the first information, and the second information in a case in which the user is present in a third region which is located across the first region and a second region that is outside the first region and is adjacent to the first region or a third region located between the first region and a second region that is located further than the first region when the image is captured (Fig. 4, ¶¶53-56, “function setting unit 22 limits the gestures identifiable by the gesture identification unit 23 when the distance D0 is greater than the first set distance D1 and less than or equal to a second set distance D2”).
Tang-Goulding-Anezaki-Savarit and Takayanagi are analogous art to the claimed invention because they are from a similar field of endeavor of detecting operator gestures to execute commands. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-Anezaki-Savarit resulting in resolutions as disclosed by Takayanagi with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-Anezaki-Savarit as described above to provide higher accuracy in association with an acceptable number of recognizable gesture (Takayanagi, ¶10, “identifiable gestures are set so that the number of the identifiable gestures decreases when the operator is situated at a position far from the display unit of the information display device. As above, by the present invention, when the operator is situated at a position far from the display unit, an erroneous identification of the operator's gesture can be made unlikely to occur by excluding gestures with a small body movement from the identifiable gestures, for example”).
With regard to Claim 7,
Tang-Goulding-Anezaki-Savarit-TAKAYANAGI teach the gesture recognition system according to claim 6, wherein the one or more processors execute the instructions to recognize a gesture of the user by placing higher priority on a result of recognition based on the image and the first information than on a result of recognition based on the image and the second information in a case in which the gesture of the user is recognized on the basis of the image, the first information, and the second information (Tang, ¶22, “when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode). This is because the system has a better chance of collecting images of the operator's whole hand than collecting images of the operator's whole arm”, TAKAYANAGI, Fig. 4, ¶¶53-56, “function setting unit 22 limits the gestures identifiable by the gesture identification unit 23 when the distance D0 is greater than the first set distance D1 and less than or equal to a second set distance D2”, Tang disclose that detecting arm movement in a distance less than the long mode will be difficult therefore the gesture will be based on hand). The same motivation to combine for claim 6 equally applies for current claim.
Claims 10 are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. [US 2019/0155313 A1, hereinafter Tang] in view of Goulding [US 2011/0231050 A1] in view of Anezaki [US 2010/0222925 A1] in view of Savarit et al. [US 2018/0012502 A1, hereinafter Savarit] in view of https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8367759 [hereinafter D1] further in view of Mao et al. [WO 2017/114941, hereinafter D2]
With regard to Claim 10,
Tang-Goulding-Anezaki-Savarit teach the mobile object according to claim 9, a second imager configured to image a user who remotely operates the mobile object (¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator”). The same motivation to combine for claim 1 equally applies for current claim.
Tang-Goulding-Anezaki-Savarit does not teach a first imager configured to image surroundings of the mobile object, wherein the one or more processors execute the instructions to attempt processing for recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager and employ, with higher priority, a result of the recognition based on the second image than a result of the recognition on the basis of the first image and cause the mobile object to be controlled on the basis of a surrounding situation obtained from the image captured by the first imager and the operation associated with the gesture recognized by the recognizer.
D1 teach a first imager configured to image surroundings of the mobile object (Fig. 2, Abstract, “Computer vision-based methods rely on the ability of a drones camera to capture surrounding images and use pattern recognition to translate images to meaningful and/or actionable information”, 3.3, “is imperative for the drone to carry out the action while ensuring safety to itself, surrounding objects, and the environment. Collision to any of these entities potentially causes serious damage to the parties involved, which is highly undesirable. In our framework, action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”); and
a second imager configured to image a user who remotely operates the mobile object (Fig. 2, Abstract, “image segregation from the video streams of front camera, creating a robust and reliable image recognition based on segregated images, and finally conversion of classified gestures into actionable drone movement, such as takeoff, landing, hovering and so forth”),
wherein the one or more processors execute the instructions to attempt processing for recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager and employ, with higher priority, a result of the recognition based on the second image than a result of the recognition on the basis of the first image (Fig. 2, Abstract, “Computer vision-based methods rely on the ability of a drones camera to capture surrounding images and use pattern recognition to translate images to meaningful and/or actionable information”, 3.3, “After a gesture is recognized and converted to a command, such as move to the left, the action planner on the drone kicks in to compute the most appropriate course of action that satisfies the recent command. In this process, it is imperative for the drone to carry out the action while ensuring safety to … action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”, Fig. 2, Abstract, “image segregation from the video streams of front camera, creating a robust and reliable image recognition based on segregated images, and finally conversion of classified gestures into actionable drone movement, such as takeoff, landing, hovering and so forth”, system provide priority to the second image that capture user’s gesture and transfer it to a command before using the first image by the action planner to identify the safest form to execute the command), and
cause the mobile object to be controlled on the basis of a surrounding situation obtained from the image captured by the first imager and the operation associated with the gesture recognized by the recognizer (3.3, “After a gesture is recognized and converted to a command, such as move to the left, the action planner on the drone kicks in to compute the most appropriate course of action that satisfies the recent command. In this process, it is imperative for the drone to carry out the action while ensuring safety to … action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”, Fig. 2).
Tang-Goulding-Anezaki-Savarit and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of detecting operator gestures to execute commands. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-Anezaki-Savarit resulting in resolutions as disclosed by D1 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-Anezaki-Savarit as described above to improve the safety of operating movable objects by taking the surrounding environment into consideration when controlling the movable object (D1, 3.3, “After a gesture is recognized and converted to a command, such as move to the left, the action planner on the drone kicks in to compute the most appropriate course of action that satisfies the recent command. In this process, it is imperative for the drone to carry out the action while ensuring safety to … action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”, Fig. 2)
Examiner notes that the claim does not disclose that the first imager is different than the second imager as they could be the same device in different instances (i.e. first imager when the device capture an image of the surrounding and second imager when capturing image of a user).
However, in effort to expedite prosecution D2 disclose the ability to use a first imagery to image surroundings and a second imagery different than the first imager to image a user.
Tang-Goulding-Anezaki-Savarit-D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of detecting and identifying objects using captured images. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Anezaki-D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-Anezaki-Savarit-D1 as described above to provide more flexibility and range for controlling moveable objects as the different imagery devices allow the capture and analysis of user gestures or commands from any location due to the fact the second imagery device that monitor the user gestures is not part of the moveable object. This is a use of known technique to improve similar devices (methods, or products) in the same way and Applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Claims 12 are rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. [US 2019/0155313 A1, hereinafter Tang] in view of Goulding [US 2011/0231050 A1] in view of Anezaki [US 2010/0222925 A1] in view of Savarit et al. [US 2018/0012502 A1, hereinafter Savarit] in view of ZHOU et al . [US 2021/0294423 A1, hereinafter ZHOU]
With regard to Claim 12,
Tang-Goulding-Anezaki-Savarit teach the mobile object according to claim 9, wherein the one or more processors execute the instructions, control the mobile object on the basis of the gesture of the user who is being tracked (Fig. 2, ¶37, “collected images and the measured distance information can be related or associated such that the system 200 can utilize both to identify (and, optionally, verify) a gesture by the operator”, ¶¶38-39. ¶41, “verification component 209 can analyze the stored images to see if the identified gesture in these images is consistent from one frame to another. If so, the system 200 can move forward with the instruction generation component 211 forming a corresponding instruction based on the identified gesture”).
Tang-Goulding-Anezaki-Savarit does not explicitly teach track a user as a target on the basis of a captured image, recognize a gesture of the user who is being tracked, and not perform processing for recognizing gestures of persons who are not being tracked.
ZHOU teach track a user as a target on the basis of a captured image, recognize a gesture of the user who is being tracked, and not perform processing for recognizing gestures of persons who are not being tracked (¶87, “use the priority ruleset to identify a user having priority, called a primary user or master user, thereby causing the face detection and tracking subsystem 312, virtual gesture-space generation subsystem 314, and hand detection and tracking subsystem 316 to only detect and track that user until priority shifts to a different user”, ¶¶163-164, “priority ruleset 352 is a designated-master-user ruleset. The designated-master-user ruleset operates by identifying a single primary user, called the master user, and monitoring tracking the single master user for gesture controls”), and
control an object on the basis of the gesture of the user who is being tracked (¶87, “use the priority ruleset to identify a user having priority, called a primary user or master user, thereby causing the face detection and tracking subsystem 312, virtual gesture-space generation subsystem 314, and hand detection and tracking subsystem 316 to only detect and track that user until priority shifts to a different user”, ¶¶163-164, “priority ruleset 352 is a designated-master-user ruleset. The designated-master-user ruleset operates by identifying a single primary user, called the master user, and monitoring tracking the single master user for gesture controls”).
Tang-Goulding-Anezaki-Savarit and ZHOU are analogous art to the claimed invention because they are from a similar field of endeavor of identifying and using hand gesture. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-Anezaki-Savarit resulting in resolutions as disclosed by ZHOU with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-Anezaki-Savarit as described above to save computing resources required for recognizing hand gestures in digital images that include multiple users rather than a single user, and may result in confusion in recognizing hand gestures performed by multiple users and making efficient use of computing resources, minimize confusion, and resolve interactions between hand gestures performed by more than one user (ZHOU, ¶¶3-4).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. [US 2019/0155313 A1, hereinafter Tang] in view of Goulding [US 2011/0231050 A1] in view of https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8367759 [hereinafter D1] further in view of Mao et al. [WO 2017/114941, hereinafter D2] in view of ZHOU et al . [US 2021/0294423 A1, hereinafter ZHOU]
With regard to Claim 16,
Tang teach a gesture recognition system comprising:
a storage device configured to store instructions (Fig. 2, 202); and
one or more processors, wherein the one or more processors execute the instructions stored in the storage device to (Fig. 2, 201, ¶7, ¶30)
acquire an image capturing a user (¶7, “programming a computer-readable medium with instructions that, when executed, can generate an image corresponding to an operator”, ¶5, “image-collection component can be a color-sensing camera that can be used to collect color images (e.g., those having red, green, and blue (RGB) pixels.)”),
recognize a region where the user is present when the image is captured(¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator”), and
in a case in which the user is present in a first region when the image is captured, recognize a gesture of the user on the basis of the image and first information for recognizing the gesture of the user (¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator. The instructions can identify a portion of the image corresponding to the operator and retrieve a second set of distance information from the first set of distance information, based at least in part on the identified portion of the image corresponding to the operator”, ¶22, “One factor to consider when determining a type of the preferred gesture is the distance between the moveable device and the operator. For example, when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode)”), wherein the gesture of the user generated by the user initiates a movement of a mobile object (¶6. “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction”);, and
in a case in which the user is present in a second region when the image is captured, recognize a gesture of the user on the basis of the image and second information for recognizing the gesture of the user (¶7, ¶22, “when the distance between the moveable device and the operator is greater than the pre-determined range, the system can select an “arm gesture” as the preferred gesture (i.e., in a long distance mode”), a storage device configured to store reference information in which a gesture of the user and an operation of the mobile object are associated (Tang, Fig. 2, ¶21, “associated system can compare the collected images and measured distance information with reference information or a pre-determined dataset stored in a pre-existing database (e.g., can be a remote database or data storage in the system)”, ¶37, “collected images and the measured distance information can be related or associated such that the system 200 can utilize both to identify (and, optionally, verify) a gesture by the operator”, ¶¶38-39. ¶41, “verification component 209 can analyze the stored images to see if the identified gesture in these images is consistent from one frame to another. If so, the system 200 can move forward with the instruction generation component 211 forming a corresponding instruction based on the identified gesture”, “verification component 209 can verify the identified gesture by comparing it with reference information (e.g., a reference set of hand gesture depth information from multiple operators) located in a (remote or local) database”),
control the mobile object on the basis of the operation of the mobile object associated with the gesture of the user with reference to the reference information (Tang, Fig. 2, ¶30, ¶37, “collected images and the measured distance information can be related or associated such that the system 200 can utilize both to identify (and, optionally, verify) a gesture by the operator”, ¶¶38-39. ¶41, “verification component 209 can analyze the stored images to see if the identified gesture in these images is consistent from one frame to another. If so, the system 200 can move forward with the instruction generation component 211 forming a corresponding instruction based on the identified gesture”)
Tang does not explicitly teach wherein
Goulding wherein the gesture of the user and a defined sound generated by the user initiates [command], and recognizing the gesture and the defined sound (¶465, “High-level control is input via external commands through the network interface 95, through body language of the at least one rider or passenger 101 as measured by the force and torque sensors 81 and IMU 82, an external control device, such as for example a radio control unit, voice commands, or visual commands or gestures, or any combination of such devices and sensing”, ¶483, ¶487, “ridden by a human operator that communicates with the robot via body motions, verbal commands, and an interface (e.g., handle bars and hand-grip controls)”).
Tang and Goulding are analogous art to the claimed invention because they are from a similar field of endeavor of controlling mobile devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang resulting in resolutions as disclosed by Goulding with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang as described above to provide the ability to Mimics how people naturally communicate, making technology feel more responsive and less demanding, empowers users with mobility or visual impairments to control devices independently, offering alternatives to traditional keyboards/mice, improve productivity by allowing users to perform complex tasks more quickly, and provide contextual understanding as gestures (like pointing) add spatial context to voice commands, clarifying intent. This is simply use of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Tang-Goulding does not explicitly disclose a first imager configured to image surroundings of the mobile object; a second imager configured to image a user who remotely operates the mobile object, attempt processing for recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager and employ, with higher priority, a result of the recognition based on the second image than a result of the recognition on the basis of the first image, and cause the mobile object to be controlled on the basis of a surrounding situation obtained from the image captured by the first imager and the operation associated with the gesture recognized.
D1 teach a first imager configured to image surroundings of the mobile object (Fig. 2, Abstract, “Computer vision-based methods rely on the ability of a drones camera to capture surrounding images and use pattern recognition to translate images to meaningful and/or actionable information”, 3.3, “is imperative for the drone to carry out the action while ensuring safety to itself, surrounding objects, and the environment. Collision to any of these entities potentially causes serious damage to the parties involved, which is highly undesirable. In our framework, action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”);
a second imager configured to image a user who remotely operates the mobile object (Fig. 2, Abstract, “image segregation from the video streams of front camera, creating a robust and reliable image recognition based on segregated images, and finally conversion of classified gestures into actionable drone movement, such as takeoff, landing, hovering and so forth”), and ,
attempt processing for recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager and employ, with higher priority, a result of the recognition based on the second image than a result of the recognition on the basis of the first image (Fig. 2, Abstract, “Computer vision-based methods rely on the ability of a drones camera to capture surrounding images and use pattern recognition to translate images to meaningful and/or actionable information”, 3.3, “After a gesture is recognized and converted to a command, such as move to the left, the action planner on the drone kicks in to compute the most appropriate course of action that satisfies the recent command. In this process, it is imperative for the drone to carry out the action while ensuring safety to … action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”, Fig. 2, Abstract, “image segregation from the video streams of front camera, creating a robust and reliable image recognition based on segregated images, and finally conversion of classified gestures into actionable drone movement, such as takeoff, landing, hovering and so forth”, system provide priority to the second image that capture user’s gesture and transfer it to a command before using the first image by the action planner to identify the safest form to execute the command), and
cause the mobile object to be controlled on the basis of a surrounding situation obtained from the image captured by the first imager and the operation associated with the gesture recognized (3.3, “After a gesture is recognized and converted to a command, such as move to the left, the action planner on the drone kicks in to compute the most appropriate course of action that satisfies the recent command. In this process, it is imperative for the drone to carry out the action while ensuring safety to … action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”, Fig. 2).
Tang-Goulding and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of detecting operator gestures to execute commands. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding resulting in resolutions as disclosed by D1 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding as described above to improve the safety of operating movable objects by taking the surrounding environment into consideration when controlling the movable object (D1, 3.3, “After a gesture is recognized and converted to a command, such as move to the left, the action planner on the drone kicks in to compute the most appropriate course of action that satisfies the recent command. In this process, it is imperative for the drone to carry out the action while ensuring safety to … action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”, Fig. 2)
Examiner notes that D1 does not disclose that the first imager is different than the second imager as they are the same device in different instances for D1 (i.e. first imager when the device capture an image of the surrounding and second imager when capturing image of a user).
Tang-Goulding-D1 does not explicitly disclose first imager is different with the second imager.
D2 teach a first imager is different with the second imager (¶4, “a camera for monitoring at least one of the event, the activity, or the status of an environment … a second camera configured to track a change in biometric position of a person … the first and second camera are a video camera, an analog camera, a digital video camera, a color camera, a monochrome camera, a camcorder, a PC camera, a webcam, and a CCTV …”).
Tang-Goulding-D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of detecting and identifying objects using captured images. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-D1 as described above to provide more flexibility and range for controlling moveable objects as the different imagery devices allow the capture and analysis of user gestures or commands from any location due to the fact the second imagery device that monitor the user gestures is not part of the moveable object.
Tang-Goulding-D1-D2 does not explicitly disclose the second camera is capable of controlling the imaging direction using a machine mechanism, captures an image around the user as a tracking target at the center.
ZHOU teach the second camera is capable of controlling the imaging direction using a machine mechanism, captures an image around the user as a tracking target at the center (claim 12, ZHOU et al . [US 2021/0294423 A1, hereinafter ZHOU] ¶87, “use the priority ruleset to identify a user having priority, called a primary user or master user, thereby causing the face detection and tracking subsystem 312, virtual gesture-space generation subsystem 314, and hand detection and tracking subsystem 316 to only detect and track that user until priority shifts to a different user”, ¶¶163-164, “priority ruleset 352 is a designated-master-user ruleset. The designated-master-user ruleset operates by identifying a single primary user, called the master user, and monitoring tracking the single master user for gesture controls”).
Tang-Goulding-D1-D2 and ZHOU are analogous art to the claimed invention because they are from a similar field of endeavor of identifying and using hand gesture. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-D1-D2 resulting in resolutions as disclosed by ZHOU with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-D1-D2 as described above to save computing resources required for recognizing hand gestures in digital images that include multiple users rather than a single user, and may result in confusion in recognizing hand gestures performed by multiple users and making efficient use of computing resources, minimize confusion, and resolve interactions between hand gestures performed by more than one user (ZHOU, ¶¶3-4).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Tang et al. [US 2019/0155313 A1, hereinafter Tang] in view of Goulding [US 2011/0231050 A1] in view of https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8367759 [hereinafter D1] further in view of Mao et al. [WO 2017/114941, hereinafter D2] in view of ZHOU et al . [US 2021/0294423 A1, hereinafter ZHOU] in view of “Multiple-Hand-Gesture Tracking using Multiple Cameras” [hereinafter D3]
With regard to Claim 17,
Tang teach a gesture recognition system comprising:
a storage device configured to store instructions (Fig. 2, 202); and
one or more processors, wherein the one or more processors execute the instructions stored in the storage device to (Fig. 2, 201, ¶7, ¶30)
acquire an image capturing a user (¶7, “programming a computer-readable medium with instructions that, when executed, can generate an image corresponding to an operator”, ¶5, “image-collection component can be a color-sensing camera that can be used to collect color images (e.g., those having red, green, and blue (RGB) pixels.)”),
recognize a region where the user is present when the image is captured(¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator”), and
in a case in which the user is present in a first region when the image is captured, recognize a gesture of the user on the basis of the image and first information for recognizing the gesture of the user (¶7, “generate an image corresponding to an operator and a first set of distance information corresponding to the operator. The instructions can identify a portion of the image corresponding to the operator and retrieve a second set of distance information from the first set of distance information, based at least in part on the identified portion of the image corresponding to the operator”, ¶22, “One factor to consider when determining a type of the preferred gesture is the distance between the moveable device and the operator. For example, when the distance between the moveable device and the operator is within a pre-determined range (e.g., 0.5-1 meter) or less than a threshold value (e.g., 2 meters), the system can select a “hand gesture” as the preferred gesture (i.e., in a short distance mode)”) wherein the gesture of the user generated by the user initiates a movement of a mobile object (¶6. “associated system can analyze these identified gestures and associate them with corresponding instructions that can be used to control the moving device. For example, the operator can instruct the moveable device to move in a certain direction by positioning his or her arm in the same direction”), and
in a case in which the user is present in a second region when the image is captured, recognize a gesture of the user on the basis of the image and second information for recognizing the gesture of the user (¶7, ¶22, “when the distance between the moveable device and the operator is greater than the pre-determined range, the system can select an “arm gesture” as the preferred gesture (i.e., in a long distance mode”), a storage device configured to store reference information in which a gesture of the user and an operation of the mobile object are associated (Tang, Fig. 2, ¶21, “associated system can compare the collected images and measured distance information with reference information or a pre-determined dataset stored in a pre-existing database (e.g., can be a remote database or data storage in the system)”, ¶37, “collected images and the measured distance information can be related or associated such that the system 200 can utilize both to identify (and, optionally, verify) a gesture by the operator”, ¶¶38-39. ¶41, “verification component 209 can analyze the stored images to see if the identified gesture in these images is consistent from one frame to another. If so, the system 200 can move forward with the instruction generation component 211 forming a corresponding instruction based on the identified gesture”, “verification component 209 can verify the identified gesture by comparing it with reference information (e.g., a reference set of hand gesture depth information from multiple operators) located in a (remote or local) database”),
control the mobile object on the basis of the operation of the mobile object associated with the gesture of the user with reference to the reference information (Tang, Fig. 2, ¶30, ¶37, “collected images and the measured distance information can be related or associated such that the system 200 can utilize both to identify (and, optionally, verify) a gesture by the operator”, ¶¶38-39. ¶41, “verification component 209 can analyze the stored images to see if the identified gesture in these images is consistent from one frame to another. If so, the system 200 can move forward with the instruction generation component 211 forming a corresponding instruction based on the identified gesture”).
Tang does not explicitly teach wherein
Goulding wherein the gesture of the user and a defined sound generated by the user initiates [command], and recognizing the gesture and the defined sound (¶465, “High-level control is input via external commands through the network interface 95, through body language of the at least one rider or passenger 101 as measured by the force and torque sensors 81 and IMU 82, an external control device, such as for example a radio control unit, voice commands, or visual commands or gestures, or any combination of such devices and sensing”, ¶483, ¶487, “ridden by a human operator that communicates with the robot via body motions, verbal commands, and an interface (e.g., handle bars and hand-grip controls)”).
Tang and Goulding are analogous art to the claimed invention because they are from a similar field of endeavor of controlling mobile devices. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang resulting in resolutions as disclosed by Goulding with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang as described above to provide the ability to Mimics how people naturally communicate, making technology feel more responsive and less demanding, empowers users with mobility or visual impairments to control devices independently, offering alternatives to traditional keyboards/mice, improve productivity by allowing users to perform complex tasks more quickly, and provide contextual understanding as gestures (like pointing) add spatial context to voice commands, clarifying intent. This is simply use of known technique to improve similar devices (methods, or products) in the same way and applying a known technique to a known device (method, or product) ready for improvement to yield predictable results (MPEP 2143).
Tang-Goulding does not explicitly disclose a first imager configured to image surroundings of the mobile object; a second imager configured to image a user who remotely operates the mobile object.
D1 teach a first imager configured to image surroundings of the mobile object (Fig. 2, Abstract, “Computer vision-based methods rely on the ability of a drones camera to capture surrounding images and use pattern recognition to translate images to meaningful and/or actionable information”, 3.3, “is imperative for the drone to carry out the action while ensuring safety to itself, surrounding objects, and the environment. Collision to any of these entities potentially causes serious damage to the parties involved, which is highly undesirable. In our framework, action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”);
a second imager configured to image a user who remotely operates the mobile object (Fig. 2, Abstract, “image segregation from the video streams of front camera, creating a robust and reliable image recognition based on segregated images, and finally conversion of classified gestures into actionable drone movement, such as takeoff, landing, hovering and so forth”).
Tang-Goulding and D1 are analogous art to the claimed invention because they are from a similar field of endeavor of detecting operator gestures to execute commands. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang resulting in resolutions as disclosed by D1 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang as described above to improve the safety of operating movable objects by taking the surrounding environment into consideration when controlling the movable object (D1, 3.3, “After a gesture is recognized and converted to a command, such as move to the left, the action planner on the drone kicks in to compute the most appropriate course of action that satisfies the recent command. In this process, it is imperative for the drone to carry out the action while ensuring safety to … action planning module requires the drone to utilize its sensors (e.g., camera and proximity) to estimate the area where it can safely fly or hover to”, Fig. 2)
Examiner notes that D1 does not disclose that the first imager is different than the second imager as they are the same device in different instances for D1 (i.e. first imager when the device capture an image of the surrounding and second imager when capturing image of a user).
Tang-Goulding-D1 does not explicitly disclose first imager is different with the second imager.
D2 teach a first imager is different with the second imager (¶4, “a camera for monitoring at least one of the event, the activity, or the status of an environment … a second camera configured to track a change in biometric position of a person … the first and second camera are a video camera, an analog camera, a digital video camera, a color camera, a monochrome camera, a camcorder, a PC camera, a webcam, and a CCTV …”).
Tang-Goulding-D1 and D2 are analogous art to the claimed invention because they are from a similar field of endeavor of detecting and identifying objects using captured images. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-D1 resulting in resolutions as disclosed by D2 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-D1 as described above to provide more flexibility and range for controlling moveable objects as the different imagery devices allow the capture and analysis of user gestures or commands from any location due to the fact the second imagery device that monitor the user gestures is not part of the moveable object.
Tang-Goulding-D1-D2 does not explicitly disclose the second camera is capable of controlling the imaging direction using a machine mechanism, captures an image around the user as a tracking target at the center.
ZHOU teach the second camera is capable of controlling the imaging direction using a machine mechanism, captures an image around the user as a tracking target at the center (claim 12, ZHOU et al . [US 2021/0294423 A1, hereinafter ZHOU] ¶87, “use the priority ruleset to identify a user having priority, called a primary user or master user, thereby causing the face detection and tracking subsystem 312, virtual gesture-space generation subsystem 314, and hand detection and tracking subsystem 316 to only detect and track that user until priority shifts to a different user”, ¶¶163-164, “priority ruleset 352 is a designated-master-user ruleset. The designated-master-user ruleset operates by identifying a single primary user, called the master user, and monitoring tracking the single master user for gesture controls”).
Tang-Goulding-D1-D2 and ZHOU are analogous art to the claimed invention because they are from a similar field of endeavor of identifying and using hand gesture. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-Goulding-D1-D2 resulting in resolutions as disclosed by ZHOU with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-D1-D2 as described above to save computing resources required for recognizing hand gestures in digital images that include multiple users rather than a single user, and may result in confusion in recognizing hand gestures performed by multiple users and making efficient use of computing resources, minimize confusion, and resolve interactions between hand gestures performed by more than one user (ZHOU, ¶¶3-4).
Tang-Goulding-D1-D2-ZHOU does not explicitly teach in a case that recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager
D3 teach in a case that recognizing a gesture of the user on the basis of a first image captured by the first imager and a second image captured by the second imager when the user is in the first region, control the mobile object on the basis of the gesture of the user on the basis of the second image captured (Fig. 11, P.476, “After the hand posture has been determined, the image with the most perpendicular axis to the hand plane is selected for the hands from available (nonoccluded) images”, “The best camera is selected based on the hand rotation angle, and the selected viewpoint varies according to hand rotation”),
in a case that recognizing a gesture of the user on the basis of a first image captured by the first imager and not recognizing the gesture of the user on the basis of a second image captured by the second imager when the user is in the first region, control the mobile object on the basis of the gesture of the user on the basis of the first image captured (“After the hand posture has been determined, the image with the most perpendicular axis to the hand plane is selected for the hands from available (non-occluded) images”, using only non-occluded images show that occluded or unusable images (images that don’t show the gesture input) are excluded).
Tang-Goulding-D1-D2-ZHOU and D3 are analogous art to the claimed invention because they are from a similar field of endeavor of identifying and using hand gesture. Thus, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Tang-D1-D2-ZHOU resulting in resolutions as disclosed by D3 with a reasonable expectation of success.
One of ordinary skill in the art would be motivated to modify Tang-Goulding-D1-D2-ZHOU as described above to provide a viewpoint selection mechanism that can effectively reduce both self-occlusion and hand-hand occlusion problems without requiring a detailed 3-D modeling and also reduces the computational costs for both the model construction and its reconstruction (D3, P.478, 6, Conclusion).
Response to Arguments
Applicant argue that the amendments are supported and no new matter is added.
Examiner respectfully disagrees, Claims 1-7, 9-10, 12-13, 15-17 contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Independent claims require to initiate a movement of a mobile object based on a combination of sound and gesture commands. Examiner could not find support for such combination. The specification disclose the ability to process commands based on gesture, sound, touch panel, OR terminal input. For example the specification of the published application ¶66 disclose that commands may come from gesture or sound “based on gesture or sound of a user” and not both combines, ¶91-97 disclose registration intention based on three independent alternatives gesture OR sound OR touch not a combination, and finally ¶¶140-148 show the ability to use gestures for control without the usage of any sound commands and ¶82 is not related to any gesture or sound combination. The dependent claims inherit the deficiencies of the independent claims.
The current amendments for claims 16 and 17 does not overcome the previous 35 USC 112(b) rejection.
Applicant argue that independent claims 1, 13, and 14 cannot be applied in the mind as they require a movement of an object using a processor.
Examiner respectfully disagrees,
First, recognizing a gesture and determining a trajectory for movement based on the detected gesture and sound is a mental process as user is able to identify a gesture associated with sound and provide command based on the combination of the detected gesture and sound. In addition a human can determine an associated path for movement based on the recognized combination. Additionally, a user can manually move an object based on the identified trajectory.
Second, using a processor do not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer, “the claim invokes computers or other machinery merely as a tool to perform an existing process”, MPEP 2106.05(f)(2), and looking to MPEP 2106.05 (d), based on court decisions well understood, routine and conventional computer functions.
Third, examiner notes that claim 14 does not include the argued limitations of using a combined input of sound and gesture to initiate a movement of an object.
Applicant’s arguments with respect to claims 1 and 13 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant argue that the combination of Tang et al., Anezaki, and Savarit et al. also fails to teach or suggest: the trajectory is further generated by connecting a first arc and a second arc of the group of arcs, and the first arc and the second arc are trajectories that satisfy a group of conditions comprising: a forward direction of the mobile object is a X direction, a X line is an imaginary line extending in the X direction, a lateral direction of the mobile object is a Y direction, a Y line is imaginary line perpendicularly intersecting the X direction and extending in the Y direction, a first end point of the first arc is Zm1, a tangential line representing Zm1 is a first tangential line, a second end point of the second arc is Zm2, a tangential line representing Zm2 is a second tangential line, a first angle formed by a first line perpendicularly intersecting the first tangential line and extending in the Y direction and the Y line is represented as 81, a second angle formed by a second line perpendicularly intersecting the second tangential line and extending in the Y direction and the first line is represented as 82, and a third angle formed by the second line and the X line is 81+82, as recited in amended independent claim 14.
Examiner respectfully disagrees, Goulding wherein a forward direction of the mobile object is a X direction, a X line is an imaginary line extending in the X direction (¶436, “FIG. 4 illustrates the side skeletal view … wherein the x and z axes with the side or y axis into and out of the page. By convention, the left side is the forward direction”, ¶324, “propels the body in the x, y, and z axes”, ¶433, ” positioned in six dimensions (X-Y-Z and roll-pitch-yaw axis, respectively)”, ¶435, “center of gravity 58 and its projection to the ground 55, called the center of pressure 59”), a lateral direction of the mobile object is a Y direction (¶436, “FIG. 4 illustrates the side skeletal view … wherein the x and z axes with the side or y axis into and out of the page” ¶324, “propels the body in the x, y, and z axes”, ¶433, ” positioned in six dimensions (X-Y-Z and roll-pitch-yaw axis, respectively)”), a Y line is imaginary line perpendicularly intersecting the X direction and extending in the Y direction (¶436, “FIG. 4 illustrates the side skeletal view … wherein the x and z axes with the side or y axis into and out of the page” ¶324, “propels the body in the x, y, and z axes”, ¶433, ” positioned in six dimensions (X-Y-Z and roll-pitch-yaw axis, respectively)”). And Savarit teach determining a trajectory along which the mobile object moves from a first trajectory point to a second trajectory point (¶15, “flight plan comprises an ordered series of segments … trajectory arising from the flight plan PV is constructed gradually from segment to segment on the basis of the directives contained in each segment (inter-waypoint geometry defined by these segments)”, ¶17, “Each segment thus generates a portion of trajectory or elementary trajectory … a straight section, an arc, typically a circular arc, or combinations of straight section and arcs”, ¶21), wherein the trajectory between the first trajectory point and the second trajectory point is determined based on a conversion of coordinate representations of the first trajectory point and the second trajectory point into the camera coordinate system data representative of the first trajectory point and the second trajectory point (¶15, “flight plan comprises an ordered series of segments … trajectory arising from the flight plan PV is constructed gradually from segment to segment on the basis of the directives contained in each segment (inter-waypoint geometry defined by these segments)”, ¶17, “Each segment thus generates a portion of trajectory or elementary trajectory … a straight section, an arc, typically a circular arc, or combinations of straight section and arcs”, ¶102, “module PRED a vertical trajectory as a function typically of a curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement, taking account of the speed of the aircraft)”, , ¶132, ¶¶104-105, ¶109, system allow and use conversion of coordinate representations of the first trajectory point and the second trajectory point between different coordinates as geometry, arc, angular curvilinear abscissa, etc. ), wherein the trajectory is further determined by combining an arc of a group of arcs representative of the trajectory, wherein the arc of the group of arcs has a different curvature radii from other arcs comprising the group of arcs (¶92, “making it possible to construct a “multi-radii” transition”, ¶109, “arc of the improved transition, that is to say its curved part, is composed of several circular arcs of radius R(i) which follow one another in a continuous and ordered manner for i=1 to N”, ¶110, “arc consisting of 3 circular arcs of increasing respective radii R1, R2 and R3”, ¶112, ¶115, “Timp constructed comprises a first arc consisting of two circular arcs respectively of radii R1, R2 … and of a second arc consisting of two circular arcs, respectively of radii R3 and R4”, Fig. 7-8), wherein the arc of the group arcs is associated with a predicted time period of a sequence of predicted time periods (¶19, “On the basis of the complete calculation of the trajectory, the FMS determines “predictions” … that is to say for various values of the curvilinear abscissa x of the trajectory”, ¶102, “curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement, taking account of the speed of the aircraft). The predictions … are the values taken by these parameters at certain curvilinear abscissae of the trajectory”, ¶104, “for each subdivision Sub(i), an associated turning radius R(i) is determined, calculated on the basis of a representative value PA(i), PB(i), PC(i), PD(i) of each parameter for the subdivision i, extrapolated on the basis of the predicted values, determined in step 3”, each subdivision (arc segment) use prediction based on x that is distance or time elapsed, therefore each arc is associated with a predicted (elapsed) time window (sequence of predicted time periods in the subdivisions). This produce predicted sequence of intervals (predicted time periods), and this identify position via X), and wherein the predicted time period indicates a defined position on the trajectory along which the mobile object moves (¶102, “curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory (or a time elapsed from the commencement …), ¶141, “Typically x is the curvilinear abscissa of the transition”), and
Initiating the movement of the mobile object in accordance with the trajectory determined based on the conversion of coordinate representations of the first trajectory point and the second trajectory point (¶20, “FMS can, on instruction from the pilot, slave the aircraft automatically to the calculated trajectory”, ¶11, “Guidance (GU ID) 107, for guiding the aircraft in the lateral and vertical planes on its three-dimensional trajectory”); and based on the trajectory determined based on the conversion of coordinate representations, causing the mobile object to move along the trajectory from the first trajectory point to the second trajectory point (Fig. 2, ¶31, “aircraft is slaved to the calculated trajectory which is frozen”, ¶20, “FMS can, on instruction from the pilot, slave the aircraft automatically to the calculated trajectory”, ¶35, “aircraft flying at nominal roll angle φN, the trajectory radius actually flown by the aircraft”, ¶117, “An improved trajectory Traj-imp incorporating the improved transition Timp is determined in step 7), and the improved trajectory Traj-imp is displayed to a pilot of the aircraft”, mobile object moves between trajectory points), wherein the trajectory is further generated by connecting a first arc and a second arc of the group of arcs, and the first arc and the second arc are trajectories that satisfy a group of conditions (¶109, “Next, in a step 6), an improved transition Timp is determined on the basis of the ordered subdivisions Sub(i) and of the successive associated turning radii R(i). The arc of the improved transition, that is to say its curved part, is composed of several circular arcs of radius R(i) which follow one another in a continuous and ordered manner for i=1 to N, N being the number of subdivisions”, ¶115, “The improved transition Timp constructed comprises a first arc consisting of two circular arcs respectively of radii R1, R2 … and of a second arc consisting of two circular arcs, respectively of radii R3 and R4”) comprising:
a forward direction of the mobile object is a X direction, a X line is an imaginary line extending in the X direction (¶43, “… a trajectory being determined on the basis of a flight plan … the trajectory being constructed gradually on the basis of the directives contained in each segment, a trajectory portion making it possible to link the elementary trajectories corresponding to two nonaligned consecutive segments being termed a transition”, ¶102, “the module PRED of the FMS. The module TRAJ calculates a lateral trajectory, and the module PRED a vertical trajectory as a function typically of a curvilinear abscissa x, which labels the position of the aeroplane on the trajectory, which position is expressed by a distance relative to the start of the trajectory”), a lateral direction of the mobile object is a Y direction, a Y line is imaginary line perpendicularly intersecting the X direction and extending in the Y direction (¶2, “ relates to a method for calculating a lateral trajectory some of whose transitions (changes of direction of the aircraft) are improved”, ¶110, “construction for an initial transition Tini between a point WPA and a heading HD. The initial transition Tini is a … circular arc of radius R0 and of angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, ¶114, “the two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, system use lateral trajectory and disclose angular sectors (i.e. angle in 2D) that require perpendicular axes), a first end point of the first arc is Zm1, a tangential line representing Zm1 is a first tangential line (¶115, “The improved transition Timp constructed comprises a first arc consisting of two circular arcs respectively of radii R1, R2 (in replacement for the first circular arc of radius R0), of a straight line portion and of a second arc”, ¶117, “An improved trajectory Traj-imp incorporating the improved transition Timp is determined in step 7), and the improved trajectory Traj-imp is displayed to a pilot of the aircraft in step 8). The trajectory is constructed gradually, so as to be continuous and differentiable”, Being differentiable at end point of first arc means a function has a well-defined, non-vertical tangent line (its best linear approximation) at that specific at end point of the first arc), a second end point of the second arc is Zm2, a tangential line representing Zm2 is a second tangential line (¶115, “and of a second arc consisting of two circular arcs, respectively of radii R3 and R4 (in replacement for the second circular arc of radius R0).”, ¶117, “… The trajectory is constructed gradually, so as to be continuous and differentiable”, Being differentiable at end point of second arc means a function has a well-defined, non-vertical tangent line (its best linear approximation) at that specific at end point of second arc), a first angle formed by a first line perpendicularly intersecting the first tangential line and extending in the Y direction and the Y line is represented as ϴ1 (¶114, “two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, ¶110, “construction for an initial transition Tini between a point WPA and a heading HD. The initial transition Tini is a … circular arc of radius R0 and of angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, ¶113, “Tini consists of a first circular arc of radius R0 (turn to the right) of angular sector equal to 135°, of a straight line portion and of a second circular arc of radius R0 (turn to the left) of angular sector equal to 45°”, direction of airplane at the beginning of the turning arc rotates as the aerplane enters the first subdivision of the transition arc (ϴ1). As the airplane turns, the tangent direction at the first end point rotate relative to Y line and the angular change occurring within this first subdivision corresponding to ϴ1, this is clarified by explicitly disclosing that the transition arc subdivided into a smaller angular sectors where the first subdivision of the subdivisions is ϴ1), a second angle formed by a second line perpendicularly intersecting the second tangential line and extending in the Y direction and the first line is represented as ϴ2 (¶114, “two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, ¶110, “construction for an initial transition Tini between a point WPA and a heading HD. The initial transition Tini is a … circular arc of radius R0 and of angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, second subdivision produces the second incremental turn angle), and a third angle formed by the second line and the X line is ϴ1 +ϴ2 (¶132, “ the criterion consists in determining a number of subdivisions N, each subdivision corresponding to an angular sector of angle equal α, the angle α corresponding to the angular sector of the initial transition θ0 divided by the number of subdivisions N”, ¶110, “angular sector 180°. It has been subdivided into 3 portions, each of angular sector equal to 60°”, ¶114, “the two circular arcs, which each correspond to a turn of the aircraft, are cut according to 2 subdivisions each, of 2 values of angular cutting sector for the 2 circular arcs of half the initial value of the angular sector, i.e. 67.5° for the first arc and 22.5° for the second”, transition turn not flown in a single big angle but in pieces, because the FMS divides the arc into sub-arcs, each with its own turning radius. So the first sub-arc produce first portion of the turn ϴ1, second sub-arc produce second portion of the turn ϴ2, third angle represent the entire turn from start to finish ϴ1+ϴ2).
As to the remaining dependent claims, applicant argue that they are allowable due to their respective direct and indirect dependencies upon one of the aforementioned Independent claims. The examiner respectfully disagrees, Independent claims were not allowable as stated in the paragraph above in this “Response to Arguments” section in this office action and there is no deficiency in the independent claims that require any of the dependent claims additional reference to be combined for teaching all elements in the independent claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to the applicant’s disclosure.
US Patent Application Publication No. 2018/0095524 filed by Chew that disclose the ability to detect and identify gestures based on captured images and different interaction modes based on the distance between the user and a machine See at least Fig. 3
Examiner has pointed out particular references contained in the prior arts of record in the body of this action for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and Figures may apply as well. It is respectfully requested from the applicant, in preparing the response, to consider fully the entire references as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior arts or disclosed by the examiner. It is noted that any citation to specific pages, columns, figures, or lines in the prior art references any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331-33, 216 USPQ 1038-39 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MOHAMED ABOU EL SEOUD whose telephone number is (303)297-4285. The examiner can normally be reached Monday-Thursday 9:00am-6:00pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MOHAMED ABOU EL SEOUD/Primary Examiner, Art Unit 2148
Prosecution Insights