Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 09/03/25 has been entered.
Response to Arguments
Applicant's arguments filed 09/03/25 have been fully considered but they are not persuasive.
Applicant’s arguments with respect to claims 1, 18 and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claims 12-13 and 18 are objected to because of the following informalities:
Claim 12 discloses in response to detecting, in the image of the physical environment, a second target object of the second target object type in the; it should be changed to in response to detecting, in the image of the physical environment, a second target object of the second target object type
Claim 13 discloses in response to detecting, in the image of the physical environment, a second target object of the second target object; it should be changed to in response to detecting, in the image of the physical environment, a second target object of the second target object type.
Claim 18 discloses detect, in the image of the physical environment, a target object of the target object at a position in the physical environment; it should be changed to detect, in the image of the physical environment, a target object of the target object type at a position in the physical environment.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 9, 11-12, 14, 17-18 and 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Waldman (US 2011/0199479) in view of Barnett et al. (US 2018/0190032).
Regarding claim 1, Waldman discloses a method (Waldman, [0005], “a method and system for displaying augmented reality maps are disclosed”) comprising:
at a device having a display (Waldman, Fig. 1), an image sensor (Waldman, [0020], “an image capture device, i.e., the camera of a smart phone”), one or more processors, and non-transitory memory (Waldman, Fig. 5);
capturing, using the image sensor, an image of a physical environment (Waldman, [0019], “a handheld communication device has captured an image 102 of the northwest corner of the intersection of Dolores St and 17th St. using its image-capturing device and displayed the image on its display”);
determining a geographic location of the physical environment (Waldman, [0024], “The geographic position of the handheld communication device can be determined using GPS coordinates or using triangulation methods using cell phone towers”);
determining a target object based on the geographic location of the device (Waldman, [0027], “The user can enter a search request for nearby points of interest based on a search term. In this example, upon entry by the user of a search for nearby "Parks" the handheld communication device sends a request for data related to nearby parks to a map database”. In addition, in paragraph [0028], “As shown in this example, the server returned points of interest "Golden Gate Park" 208, "Buena Vista Park" 206, "Midtown Terrace Playground" 210, and "Mission Dolores Park" 212. The handheld communication device determines that of the point-of-interest search results, only "Golden Gate Park" 208 and "Buena Vista Park" 206 are within the field of view of the handheld communication device”. The nearby points of interest are considered a target object);
displaying, on the display, virtual content associated with the target object (Waldman, Fig. 3).
Waldman does not expressly disclose “determining a target object type based on the geographic location”;
Barnett et al. (hereinafter Barnett) discloses determining a geographic location of a physical environment (Barnett, [0048], “if a user is detected as being in the city of San Francisco”);
determining a target object type based on the geographic location of the physical environment (Barnett, [0048], “Another machine learning model associated with a particular city may be trained to identify landmarks within that city”. Identify landmarks is considered a target object type based on the geographic location);
detecting, in an image of the physical environment, a target object of the target object type at a position in the physical environment (Barnett, [0049], “if it is detected that the user is at the Golden Gate Bridge, and/or the Golden Gate Bridge is depicted in the user's camera view”. The Golden Gate Bridge is considered a target object at a position in the physical environment);
in response to the detecting of the target object, display, on the display at the position, virtual content associated with the target object type at the position in the physical environment (Barnett, [0050], “if a user is standing on a cliff overlooking a city, the user can scan his or her camera to bring various landmarks into and out of the camera view. As a landmark enters the camera view, information about the landmark can be presented in the user's camera view.”. In addition, in paragraph [0051], “Augmented reality overlays may also be selected based on specific objects identified in a camera view, e.g., a specific landmark. For example, if the Empire State Building is depicted, an augmented reality overlay including King Kong climbing the Empire State Building may be presented to a user for potential selection, whereas if the Tokyo Tower is depicted, an augmented reality overlay including Godzilla climbing the Tokyo Tower can be presented to the user for potential selection”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to display Waldman’s virtual content using Barnett’s augmented reality overlay is determined based on the one or more objects identified in the camera view. The motivation for doing so would have been enabling users to create social media content while actively participating in real-world activities and surroundings.
Regarding claim 2, Waldman discloses a GPS signal (Waldman, [0024], “The geographic position of the handheld communication device can be determined using GPS coordinates”).
Regarding claim 3, Waldman discloses a user input (Waldman, [0039], “visually augments the captured video stream with a directional map to a selected point of interest in response to the user input…The user input can be a selection of a displayed point of interest to indicate that the user wishes to view navigation data for reaching the selected point of interest”. In addition, in paragraph [0048], “The processor 620 can also receive location and orientation information from devices such as a GPS device 632”).
Regarding claim 9, Waldman as modified by Barnett with the same motivation from claim 1 discloses displaying an animation of a virtual object (Barnett, [0051], “an augmented reality overlay in the form of an animation can be presented which makes it appear as if a large creature is rampaging through the city skyline. Augmented reality overlays can be selected based on types of objects depicted in a camera view”).
Regarding claim 11, Waldman discloses the virtual content is based on the geographic location of the physical environment (Waldman, [0005], “overlay information regarding the presently viewed objects, thus enhancing reality”).
Regarding claim 12, Waldman discloses displaying second virtual content associated with the second target object (Waldman, Fig. 1 shows visually augmented captured image with data related to a search for points of interest).
Waldman as modified by Barnett with the same motivation from claim 1 discloses determining a second target object type based on the geographic location of the physical environment (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”);
in response to detecting, in the image of the physical environment, a second target object of the second object type, displaying virtual content associated with the second target object type (Barnett, [0049], “provide one or more augmented reality overlays based on objects detected in the user's camera view for modification of the user's camera view. Augmented reality overlays can take various forms and provide various types of effects and/or modifications”).
Regarding claim 14, Waldman discloses detecting motion of the device from the physical environment to a second physical environment (Waldman, [0007], “As the user and associated device progress along a route, the overlaid directions can automatically update to show the updated path”);
determining a geographic location of the second physical environment (Waldman, [0024], “The geographic position of the handheld communication device can be determined using GPS coordinates or using triangulation methods using cell phone towers”. The user enters the name of a specific park into the smart phone at another location is considered determining a second geographic location of the device);
capturing, an image of the second physical environment (Waldman, [0010], “a user points a handheld communication device to capture and display a real-time video stream of a view”);
displaying, on the display, second virtual content associated with the second target object (Waldman, Fig. 1).
Waldman as modified by Barnett with the same motivation from claim 1 discloses determining a second target object type based on the geographic location of the second geographical environment (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”. In addition, in paragraph [0051], “if a skyline is detected in a camera view, an augmented reality overlay in the form of an animation can be presented which makes it appear as if a large creature is rampaging through the city skyline”);
detecting, in the of the second physical environment, a target object of the second target object type (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”);
in response to detecting the second target object, displaying, on the display, second virtual content associated with the second target object type (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”. In addition, in paragraph [0050], “an augmented reality overlay can provide context information about one or more objects depicted in a user's camera view”. Displaying an overlay for a recognized piece in the art museum is considered second virtual content linked to the second target object type).
Regarding claim 17, Waldman discloses displaying, on the display, third virtual content associated with the target object at the second geographic location of the device (Waldman, Fig. 1 shows displaying, on the display, third virtual content, such as street labels),
Waldman as modified by Barnett with the same motivation from claim 1 discloses in response to detecting, in the image of the second physical environment, a third target object of the target object type, displaying on the display, third virtual content associated with the target object type (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”. In addition, in paragraph [0050], “an augmented reality overlay can provide context information about one or more objects depicted in a user's camera view”.).
Regarding claim 18, Waldman discloses a device (Waldman, Fig. 3) comprising:
a display (Waldman, Fig. 5); non-transitory memory (Waldman, Fig. 5); and one or more processors (Waldman, Fig. 5).
an image sensor (Waldman, [0019], “a handheld communication device has captured an image 102 of the northwest corner of the intersection of Dolores St and 17th St. using its image-capturing device and displayed the image on its display”);
The limitations recite in claim 18 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claim 20, Waldman discloses a non-transitory memory storing one or more programs, which, when executed by one or more processors of a device (Waldman, [0053], “using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions comprise, for example, instructions and data which cause or otherwise configure a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a certain function or group of functions”) including a display (Waldman, Fig. 5).
The limitations recite in claim 20 are similar in scope to the method recited in claim 1 and therefore are rejected under the same rationale.
Regarding claim 21, Waldman as modified by Barnett with the same motivation from claim 1 discloses detecting the target object at a position in the image of the physical environment (Barnett, [0039], “if a user's camera view depicts the Golden Gate Bridge, object detection or recognition techniques can be utilized to identify the Golden Gate Bridge, and to recommend one or more augmented reality overlays based on the identified object(s)”) and displaying the image of the physical environment with the virtual content overlaid on the image of the physical environment at the position in the image of the physical environment (Barnett, [0039], “an augmented reality overlay can be recommended to a user which can include, in one instance, a post-card-style frame that is imposed on the camera view that reads “Hello from the Golden Gate Bridge!” In another example, an augmented reality overlay may include historical information about the Golden Gate Bridge presented next to the Golden Gate Bridge within the camera view”).
Regarding claim 22, Waldman as modified by Barnett with the same motivation from claim 1 discloses selecting, based on the geographic location of the physical environment, the target object type from a plurality of target object types respectively associated with a plurality of geographic locations (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city. Based on a user's location information, various machine learning models can be downloaded to the user's mobile device such that object recognition can be performed locally on the user's mobile device using the downloaded machine learning models”).
Claims 10 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Waldman (US 2011/0199479) in view of Barnett et al. (US 2018/0190032), as applied to claim 1, in further view of Goodrich et al. (US 11,227,442).
Regarding claim 10, while Waldman teaches the virtual content; Waldman as modified by Barnett does not expressly disclose “the virtual content is independent of the geographic location of the physical environment”;
Goodrich et al. (hereinafter Goodrich) discloses a virtual content is independent of a location of a physical environment (Goodrich, col 21. 48-56, “While the 3D caption with the heart emojis is presented on top of the user's face in the camera feed, as shown in FIG. 18D, the user selects an option to activate a rear-facing camera feed. In response, the camera feed is replaced by a video feed received from the rear-facing camera in which no face is detected or presented. As a result, the 3D caption system 210 presents the 3D caption that was previously on the user's head as 3D caption 1850 above the ground, as shown in FIG. 18E”. The 3D caption is considered independent of the location of the device (also see Figs. 10-15)).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Goodrich’s rendering a virtual 3D object within a camera feed in the augment reality system, as taught by Waldman. The motivation for doing so would have been automatically augment the 3D captions a user input with one or more graphical elements, as if they exist in real-world environments.
Regarding claim 15, Waldman as modified by Barnett, Hayford and Goodrich with the same motivation from claim 10 discloses the second virtual content is the same as the virtual content (Goodrich, col 21. 48-56, “While the 3D caption with the heart emojis is presented on top of the user's face in the camera feed, as shown in FIG. 18D, the user selects an option to activate a rear-facing camera feed. In response, the camera feed is replaced by a video feed received from the rear-facing camera in which no face is detected or presented. As a result, the 3D caption system 210 presents the 3D caption that was previously on the user's head as 3D caption 1850 above the ground, as shown in FIG. 18E”).
Claims 13 is rejected under 35 U.S.C. 103 as being unpatentable over Waldman (US 2011/0199479) in view of Barnett et al. (US 2018/0190032), as applied to claim 1, in further view of Goodrich et al. (US 11,227,442).
Regarding claim 13, Waldman discloses determining a second target object based the geographic location of the physical environment (Waldman, [0022], “For example, a point of interest can be a park when a user searches for nearby parks. Likewise a point of interest can be places, buildings, structures, even friends that can be located on a map, when the point of interest is searched for”. A point of interest such as places, buildings, structures, even friends determine at another location of the device is considered determining a second target object based on the geographic location of the device),
displaying, on the display, second virtual content associated with the second target object (Waldman, Fig. 1 shows visually augmented captured image with data related to a search for points of interest).
Waldman as modified by Barnett with the same motivation from claim 1 discloses determining a second target object type of the geographic location of the physical environment (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”);
in response to detecting, in the image of the physical environment, a second target object of the second target object type, displaying, on the display, second virtual content associated with the second target object type (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”. In addition, in paragraph [0050], “an augmented reality overlay can provide context information about one or more objects depicted in a user's camera view”);
Waldman as modified by Barnett does not expressly disclose “determining the second target object type independent of the geographic location”;
Hayford discloses determining a target object type independent of a geographic location (Hayford, [0039], “For example, in FIGS. 5D and 5E, the client app 306 can ask about the user's main form of transportation and how long she or he is willing to commute to work each day”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to display Waldman as modified by Barnett’s virtual content using the concept of Hayford’s collect user preferences. The motivation for doing so would have been offering users valuable insight into local product and service providers.
Claims 7 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Waldman (US 2011/0199479) in view of Barnett et al. (US 2018/0190032), as applied to claim 1, in further view of Ghazanfari (US 2019/0369742).
Regarding claim 7, Waldman as modified by Barnett does not expressly disclose “a currency”;
Ghazanfari discloses a currency (Ghazanfari, [0051], “A store of a particular clothing brand may have different appearances at different geographic locations (e.g., at different part of a country or at different countries). To provide the user with a more realistic shopping experience, it is desirable to match the appearance of the displayed virtual store to the physical store that the user is most likely to visit (e.g., a local store). For example, the language (written or spoken) and/or currency used in the virtual store can match those used in a corresponding local physical store, or the decoration and/or layout of the virtual store can match that of the local physical store”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Ghazanfari’s currency used in the virtual store can match those used in a corresponding local physical store in the augmented reality system, as taught by Waldman as modified by Barnett. The motivation for doing so would have been provide the user with a more realistic shopping experience.
Regarding claim 16, Waldman as modified by Barnett and Ghazanfari with the same motivation from claim 7 discloses a second virtual content is a localized version of the virtual content (Ghazanfari, [0051], “A store of a particular clothing brand may have different appearances at different geographic locations (e.g., at different part of a country or at different countries). To provide the user with a more realistic shopping experience, it is desirable to match the appearance of the displayed virtual store to the physical store that the user is most likely to visit (e.g., a local store). For example, the language (written or spoken) and/or currency used in the virtual store can match those used in a corresponding local physical store, or the decoration and/or layout of the virtual store can match that of the local physical store”).
Claims 23 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Waldman (US 2011/0199479) in view of Barnett et al. (US 2018/0190032), as applied to claim 22, in further view of Waldron et al. (US 2018/0150899).
Regarding claim 23, Waldman as modified by Barnett with the same motivation from claim 1 discloses a plurality of object models associated with the target object and respectively associated with a plurality of geographic locations (Barnett, [0048], “where each machine learning model is associated with a particular geographic region such that each machine learning model is trained to identify objects associated with the particular geographic region”), receiving an object model of the target object type associated with the geographic location (Barnett, [0048], “Based on a user's location information, various machine learning models can be downloaded to the user's mobile device such that object recognition can be performed locally on the user's mobile device using the downloaded machine learning models”);
Waldman as modified by Barnett discloses a database (Barnett, [0072], “a data structure including edges and nodes. Other data structures can also be used to represent the social network, including but not limited to databases”);
Waldman as modified by Barnett does not expressly disclose “submitting a query including a target object identifier and a geographic location to a database”;
Waldron et al. (hereinafter Waldron) discloses submitting a query including an object identifier and a geographic location to a database (Waldron, [0135], “Tokens 110 comprise any suitable information for requesting information from the remote server 102 and/or one or more other sources (e.g. third-party databases 118)…one or more objects 150 (e.g. one or more product identifiers 115), and the location of the user 106 (e.g. a vendor identifier 117 or a location identifier 902)”),
in response to the query, receiving information (Waldron, [0137], “receives the requested information in response to sending the token 110 and presents received information as virtual objects overlaid with tangible object in a real scene in front of the user 106”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to use the concept of Waldron’s submitting a query to a database to retrieve Waldman as modified by Barnett’s a plurality of machine learning models. The motivation for doing so would have been enabling fast storage and quick searching of large datasets.
Regarding claim 24, Waldman as modified by Barnett with the same motivation from claim 1 discloses detecting the target object is based on the object model of the target object type (Barnett, [0048], “a machine learning model associated with a particular building may be trained to identify objects within that building (e.g., a machine learning model associated with a particular art museum can be trained to identify pieces contained in the art museum). Another machine learning model associated with a particular city may be trained to identify landmarks within that city”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ZHAI whose telephone number is (571)270-3740. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached on (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KYLE ZHAI/Primary Examiner, Art Unit 2612