DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-5,7-16,18-20 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Lindmeier et al (US 20220084279 A1)
Regarding claim 1, Lindmeier discloses A human-computer interaction method ([0032] computer system enhances a two-dimensional drawing), comprising:
determining a gaze point of a line of sight of a user on a target object, wherein the target object is located in a virtual space ([0034] the user's gaze is used to determine which virtual object is the object of focus.);
adjusting, in response to an interaction gesture for the gaze point, a display form of the gaze point to generate a first interaction point ([0033] a manipulation indication is displayed, thus guiding the user to perform the requested type of manipulation); and
interacting with the target object based on the first interaction point ([0034] based on the gesture performed by the one or more hands of the user, whether the user is requesting display of a contextual menu associated with a virtual object or requesting to move the virtual object towards the user).
Regarding claim 2 Lindmeier discloses wherein the adjusting, in response to the interaction gesture for the gaze point, the display form of the gaze point to generate the first interaction point comprises:
recognizing an obtained gesture image to obtain a gesture recognition result, wherein the gesture image is acquired by an image acquisition apparatus, and the gesture image corresponds to a left hand of the user or a right hand of the user ([0080] analyze the trajectory of the hands and/or fingers over multiple frames in the sequence in order to identify gestures); and
switching, in response to that the gesture recognition result is a first gesture, the display form of the gaze point from a first form to a second form based on the first gesture, and determining the gaze point in the second form to be the first interaction point ([0080] move and modify images presented on the display generation component 120, or perform other functions, in response to the pose and/or gesture information.).
Regarding claim 3 Lindmeier discloses wherein the interacting with the target object based on the first interaction point comprises:
in response to determining that an interaction gesture acting on the first interaction point is the first gesture and the first gesture moves from a first position to a second position, controlling, based on a movement trajectory of the first gesture, the target object to move from the first position to the second position ([0090] the controller may position or move virtual content in the view based at least in part on the user's current gaze direction.).
Regarding claim 4 Lindmeier discloses wherein the interacting with the target object based on the first interaction point comprises:
in response to determining that an interaction gesture acting on the first interaction point is switched from the first gesture to a second gesture and the second gesture moves from a first position to a third position, controlling, based on a movement trajectory of the second gesture, the first interaction point to move from the first position to the third position ([0083] location and movements of these key feature points over multiple image frames are used by the controller 110 to determine the hand gestures performed by the hand or the current state of the hand,);
interacting with the target object by at least one of:
zooming in the target object in response to that the third position is on a zoom-in control and the interaction gesture on the first interaction point is switched from the second gesture to a third gesture ([0573] , the second view of the first portion of the three-dimensional environment is a zoomed in view of the first portion of the three-dimensional environment (e.g., a zoomed in view of the first portion from the same camera position as the first view, or a view from a camera position closer to the first portion of the three-dimensional environment than the camera position of the first view).);
zooming out the target object in response to that the third position is on a zoom-out control and the interaction gesture on the first interaction point is switched from the second gesture to the third gesture ([0614] if the respective user interface element is increased or decreased, the expanded view continues to display the same amount of the first portion of the three-dimensional environment (e.g., does not zoom in or zoom out, optionally only increasing or decreasing the size of the objects already being displayed); or
rotating the target object in response to that the third position is on a rotate control and the interaction gesture on the first interaction point is switched from the second gesture to the third gesture ([0195] , virtual objects are rotated in one of the three orientations based on which circular element the user is interacting with).
Regarding claim 5 Lindmeier discloses wherein the interacting with the target object based on the first interaction point comprises:
in response to that interactive voice is obtained, interacting with the target object based on the interactive voice ([0006] user interacts with the GUI through stylus and/or finger contacts and gestures on the touch-sensitive surface, movement of the user's eyes and hand in space relative to the GUI or the user's body as captured by cameras and other movement sensors, and voice inputs as captured by one or more audio input devices).
Regarding claim 7 Lindmeier discloses redisplaying the gaze point on the target object in response to that the gaze point corresponding to the line of sight of the user moves out of an observable region of the first interaction point ([0560] the “camera” for magnified view 1714 is also shifted rightwards to maintain its relative position between the “camera” for three-dimensional environment 1704 and cylinder 1708 (e.g., to remain in the line of sight for cylinder 1708).);
adjusting, in response to the interaction gesture for the gaze point, the display form of the gaze point to generate a second interaction point, wherein a real hand of the user corresponding to the second interaction point is different from a real hand of the user corresponding to the first interaction point ([0575] if the user is looking at a second portion of the three-dimensional environment when the user input is received, then display an expanded view of the second portion of the three-dimensional environment.); and
interacting with the target object based on the first interaction point and the second interaction point ([0530] while displaying the first physical object, a respective representation associated with the first physical object overlaid on the first physical object, and a selectable visual element associated with the respective representation in the three-dimensional environment (e.g., while displaying a representation of the first physical object).
Regarding claim 8 Lindmeier discloses wherein the interacting with the target object comprises: at least one of moving, zooming in, zooming out, or rotating ([0033] computing system optionally is able to determine the manipulation operation being requested by the user (e.g., movement, rotation, resizing, etc.))
Regarding claim 9 Lindmeier discloses wherein the interacting with the target object based on the first interaction point and the second interaction point comprises:
in response to determining that an interaction gesture acting on the first interaction point and an interaction gesture acting on the second interaction point both are a first gesture, controlling, based on a movement trajectory of the first gesture, the first interaction point and the second interaction point to move ([0080] software may also analyze the trajectory of the hands and/or fingers over multiple frames in the sequence in order to identify gestures. The pose estimation functions described herein may be interleaved with motion tracking functions);
moving the target object based on the first interaction point and the second interaction point in response to that both a movement variation amount and a movement direction of the first interaction point are the same as those of the second interaction point (0138] user input is a gesture or user movement detected by the one or more input devices (e.g., a hand gesture detected by a hand motion sensor in communication with the electronic device).); and
zooming in, zooming out, or rotating the target object based on the first interaction point and the second interaction point in response to that the movement directions of the first interaction point and the second interaction point are different ([0573] the second view of the first portion of the three-dimensional environment is a zoomed in view of the first portion of the three-dimensional environment (e.g., a zoomed in view of the first portion from the same camera position as the first view, or a view from a camera position closer to the first portion of the three-dimensional environment than the camera position of the first view).).
Regarding claim 10 Lindmeier discloses wherein the zooming in, zooming out, or rotating the target object based on the first interaction point and the second interaction point in response to that the movement directions of the first interaction point and the second interaction point are different comprises:
zooming in the target object based on the first interaction point and the second interaction point in response to determining that a length of a line segment between the first interaction point and the second interaction point is greater than an initial length, wherein the initial length is a length of a line segment between the first interaction point and the second interaction point before the first interaction point and the second interaction point are controlled based on the first gesture to move ([0612] the user is able to enlarge the portal by performing a pinch gesture by two hands (optionally while looking at the portal) and moving the hands to increase the distance between the two hands);
zooming out the target object based on the first interaction point and the second interaction point in response to determining that the length of the line segment between the first interaction point and the second interaction point is less than the initial length ([0614] if the respective user interface element is increased or decreased, the expanded view continues to display the same amount of the first portion of the three-dimensional environment (e.g., does not zoom in or zoom out, optionally only increasing or decreasing the size of the objects already being displayed); and
rotating the target object based on the first interaction point and the second interaction point in response to determining that the length of the line segment between the first interaction point and the second interaction point is equal to the initial length ([0195] device 101 displays a dot at a particular position on the circular element corresponding to the pitch rotation to indicate that if second hand 916-2 performs a selection input (e.g., a pinch gesture), then the pitch rotation is selected and the user is able to cause the virtual object to rotate in the pitch orientation (e.g., by moving second hand 916-2 in a circular arc in a manner indicated by the selected circular element, optionally while maintaining the selection input).).
Regarding claim 11 Lindmeier discloses in response to that the first interaction point and the second interaction point are located at a same position, optimizing the position of the second interaction point ([0211] the predetermined increments include a “snapping” functionality in which the manipulation will snap to the increment when the manipulation approaches to within a threshold distance of the snap point.).
Regarding claim 12, Lindmeier discloses an electronic device ([0032] computer system enhances a two-dimensional drawing), comprising:
a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and run the computer program stored in the memory to perform a human-computer interaction method ([0212] method 1000 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a computer system, such as the one or more processors 202 of computer system 101 (e.g., control unit 110 in FIG. 1A). Some operations in method 1000 are, optionally, combined and/or the order of some operations is, optionally, changed) comprising
determining a gaze point of a line of sight of a user on a target object, wherein the target object is located in a virtual space ([0034] the user's gaze is used to determine which virtual object is the object of focus.);
adjusting, in response to an interaction gesture for the gaze point, a display form of the gaze point to generate a first interaction point ([0033] a manipulation indication is displayed, thus guiding the user to perform the requested type of manipulation); and
interacting with the target object based on the first interaction point ([0034] based on the gesture performed by the one or more hands of the user, whether the user is requesting display of a contextual menu associated with a virtual object or requesting to move the virtual object towards the user).
Regarding claim 13 Lindmeier discloses wherein the adjusting, in response to the interaction gesture for the gaze point, the display form of the gaze point to generate the first interaction point comprises:
recognizing an obtained gesture image to obtain a gesture recognition result, wherein the gesture image is acquired by an image acquisition apparatus, and the gesture image corresponds to a left hand of the user or a right hand of the user ([0080] analyze the trajectory of the hands and/or fingers over multiple frames in the sequence in order to identify gestures); and
switching, in response to that the gesture recognition result is a first gesture, the display form of the gaze point from a first form to a second form based on the first gesture, and determining the gaze point in the second form to be the first interaction point ([0080] move and modify images presented on the display generation component 120, or perform other functions, in response to the pose and/or gesture information.).
Regarding claim 14 Lindmeier discloses wherein the interacting with the target object based on the first interaction point comprises:
in response to determining that an interaction gesture acting on the first interaction point is the first gesture and the first gesture moves from a first position to a second position, controlling, based on a movement trajectory of the first gesture, the target object to move from the first position to the second position ([0090] the controller may position or move virtual content in the view based at least in part on the user's current gaze direction.).
Regarding claim 15 Lindmeier discloses wherein the interacting with the target object based on the first interaction point comprises:
in response to determining that an interaction gesture acting on the first interaction point is switched from the first gesture to a second gesture and the second gesture moves from a first position to a third position, controlling, based on a movement trajectory of the second gesture, the first interaction point to move from the first position to the third position ([0083] location and movements of these key feature points over multiple image frames are used by the controller 110 to determine the hand gestures performed by the hand or the current state of the hand,);
interacting with the target object by at least one of:
zooming in the target object in response to that the third position is on a zoom-in control and the interaction gesture on the first interaction point is switched from the second gesture to a third gesture ([0573] , the second view of the first portion of the three-dimensional environment is a zoomed in view of the first portion of the three-dimensional environment (e.g., a zoomed in view of the first portion from the same camera position as the first view, or a view from a camera position closer to the first portion of the three-dimensional environment than the camera position of the first view).);
zooming out the target object in response to that the third position is on a zoom-out control and the interaction gesture on the first interaction point is switched from the second gesture to the third gesture ([0614] if the respective user interface element is increased or decreased, the expanded view continues to display the same amount of the first portion of the three-dimensional environment (e.g., does not zoom in or zoom out, optionally only increasing or decreasing the size of the objects already being displayed); or
rotating the target object in response to that the third position is on a rotate control and the interaction gesture on the first interaction point is switched from the second gesture to the third gesture ([0195] , virtual objects are rotated in one of the three orientations based on which circular element the user is interacting with).
Regarding claim 16 Lindmeier discloses wherein the interacting with the target object based on the first interaction point comprises:
in response to that interactive voice is obtained, interacting with the target object based on the interactive voice ([0006] user interacts with the GUI through stylus and/or finger contacts and gestures on the touch-sensitive surface, movement of the user's eyes and hand in space relative to the GUI or the user's body as captured by cameras and other movement sensors, and voice inputs as captured by one or more audio input devices).
Regarding claim 18 Lindmeier discloses redisplaying the gaze point on the target object in response to that the gaze point corresponding to the line of sight of the user moves out of an observable region of the first interaction point ([0560] the “camera” for magnified view 1714 is also shifted rightwards to maintain its relative position between the “camera” for three-dimensional environment 1704 and cylinder 1708 (e.g., to remain in the line of sight for cylinder 1708).);
adjusting, in response to the interaction gesture for the gaze point, the display form of the gaze point to generate a second interaction point, wherein a real hand of the user corresponding to the second interaction point is different from a real hand of the user corresponding to the first interaction point ([0575] if the user is looking at a second portion of the three-dimensional environment when the user input is received, then display an expanded view of the second portion of the three-dimensional environment.); and
interacting with the target object based on the first interaction point and the second interaction point ([0530] while displaying the first physical object, a respective representation associated with the first physical object overlaid on the first physical object, and a selectable visual element associated with the respective representation in the three-dimensional environment (e.g., while displaying a representation of the first physical object).
Regarding claim 19 Lindmeier discloses wherein the interacting with the target object comprises: at least one of moving, zooming in, zooming out, or rotating ([0033] computing system optionally is able to determine the manipulation operation being requested by the user (e.g., movement, rotation, resizing, etc.))
Regarding claim 20, Lindmeier discloses A non-transitory computer-readable storage medium, configured to store a computer program, wherein the computer program causes a computer to perform a human-computer interaction method ([0212] method 1000 is governed by instructions that are stored in a non-transitory computer-readable storage medium and that are executed by one or more processors of a computer system, such as the one or more processors 202 of computer system 101 (e.g., control unit 110 in FIG. 1A). Some operations in method 1000 are, optionally, combined and/or the order of some operations is, optionally, changed) comprising:
determining a gaze point of a line of sight of a user on a target object, wherein the target object is located in a virtual space ([0034] the user's gaze is used to determine which virtual object is the object of focus.);
adjusting, in response to an interaction gesture for the gaze point, a display form of the gaze point to generate a first interaction point ([0033] a manipulation indication is displayed, thus guiding the user to perform the requested type of manipulation); and
interacting with the target object based on the first interaction point ([0034] based on the gesture performed by the one or more hands of the user, whether the user is requesting display of a contextual menu associated with a virtual object or requesting to move the virtual object towards the user).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 6, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over 1, 12 in view of
Claim(s) 6 , 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lindmeier et al (US 20220084279 A1) as applied to claim 1, 12 above, and further in view of Kamhi et al (US 20220236787 A1).
Regarding claim 6 Lindmeier is discloses
moving the target object based on the voice recognition result in response to that the voice recognition result is to move the target object ([0083] location and movements of these key feature points over multiple image frames are used by the controller 110 to determine the hand gestures performed by the hand or the current state of the hand,);
zooming in the target object based on the voice recognition result in response to that the voice recognition result is to zoom in the target object ([0573] , the second view of the first portion of the three-dimensional environment is a zoomed in view of the first portion of the three-dimensional environment (e.g., a zoomed in view of the first portion from the same camera position as the first view, or a view from a camera position closer to the first portion of the three-dimensional environment than the camera position of the first view).);
zooming out the target object based on the voice recognition result in response to that the voice recognition result is to zoom out the target object ([0614] if the respective user interface element is increased or decreased, the expanded view continues to display the same amount of the first portion of the three-dimensional environment (e.g., does not zoom in or zoom out, optionally only increasing or decreasing the size of the objects already being displayed); or
rotating the target object based on the voice recognition result in response to that the voice recognition result is to rotate the target object ([0195] , virtual objects are rotated in one of the three orientations based on which circular element the user is interacting with).
Kamhi discloses wherein the interacting with the target object based on the interactive voice comprises:
recognizing the interactive voice to obtain a voice recognition result ([0035] user interaction tracking component 120 may include a voice recognition component 124 configured to recognize voice commands provided by the user in association with particular virtual articles in the rendered augmented scene); and
interacting with the target object based on the voice recognition result, comprising at least one of:
moving the target object based on the voice recognition result in response to that the voice recognition result is to move the target object ([0025] scene augmentation component 110 may be configured to dynamically track the location of the physical objects in the rendered scene to determine movement of the physical objects and cause the generated virtual articles to behave in accordance with the movement).
Lindmeier and Kamhi are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify computer system of Lindmeier to include wherein the interacting with the target object based on the interactive voice comprises: recognizing the interactive voice to obtain a voice recognition result; and interacting with the target object based on the voice recognition result, comprising at least one of: moving the target object based on the voice recognition result in response to that the voice recognition result is to move the target object as described by Kamhi.
The motivation for doing so would have been for modifying augmented reality based on user interaction with rendered augmented reality (Kamhi, [0011]).
Therefore, it would have been obvious to combine Lindmeier and Kamhi to obtain the invention as specified in claim 6.
Regarding claim 17 Lindmeier is discloses
moving the target object based on the voice recognition result in response to that the voice recognition result is to move the target object ([0083] location and movements of these key feature points over multiple image frames are used by the controller 110 to determine the hand gestures performed by the hand or the current state of the hand,);
zooming in the target object based on the voice recognition result in response to that the voice recognition result is to zoom in the target object ([0573] , the second view of the first portion of the three-dimensional environment is a zoomed in view of the first portion of the three-dimensional environment (e.g., a zoomed in view of the first portion from the same camera position as the first view, or a view from a camera position closer to the first portion of the three-dimensional environment than the camera position of the first view).);
zooming out the target object based on the voice recognition result in response to that the voice recognition result is to zoom out the target object ([0614] if the respective user interface element is increased or decreased, the expanded view continues to display the same amount of the first portion of the three-dimensional environment (e.g., does not zoom in or zoom out, optionally only increasing or decreasing the size of the objects already being displayed); or
rotating the target object based on the voice recognition result in response to that the voice recognition result is to rotate the target object ([0195] , virtual objects are rotated in one of the three orientations based on which circular element the user is interacting with).
Kamhi discloses wherein the interacting with the target object based on the interactive voice comprises:
recognizing the interactive voice to obtain a voice recognition result ([0035] user interaction tracking component 120 may include a voice recognition component 124 configured to recognize voice commands provided by the user in association with particular virtual articles in the rendered augmented scene); and
interacting with the target object based on the voice recognition result, comprising at least one of:
moving the target object based on the voice recognition result in response to that the voice recognition result is to move the target object ([0025] scene augmentation component 110 may be configured to dynamically track the location of the physical objects in the rendered scene to determine movement of the physical objects and cause the generated virtual articles to behave in accordance with the movement).
Lindmeier and Kamhi are combinable because they are from the same field of invention.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify computer system of Lindmeier to include wherein the interacting with the target object based on the interactive voice comprises: recognizing the interactive voice to obtain a voice recognition result; and interacting with the target object based on the voice recognition result, comprising at least one of: moving the target object based on the voice recognition result in response to that the voice recognition result is to move the target object as described by Kamhi.
The motivation for doing so would have been for modifying augmented reality based on user interaction with rendered augmented reality (Kamhi, [0011]).
Therefore, it would have been obvious to combine Lindmeier and Kamhi to obtain the invention as specified in claim 17.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHIVANG I PATEL whose telephone number is (571)272-8964. The examiner can normally be reached on M-F 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached on (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHIVANG I PATEL/Primary Examiner, Art Unit 2615
Prosecution Insights