Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-12 are rejected under 35 U.S.C. 103 as being unpatentable over Bronicki et al. (US Pub 2022/0383546 A1) in view of Powderly et al. (US Pub 2024/0211028 A1).
As to claim 1, Bronicki discloses a computer implemented method for generating a task to be performed in a real space (¶0003, ¶0128, ¶0222), the method comprising:
obtaining a first image captured by a camera of a first mobile electronic device being calibrated in a virtual representation of the real space such that the first mobile electronic device has a known pose in the virtual representation of the real space (¶0005, ¶0102, “the system may process images and image data acquired by a capturing device to determine information associated with products displayed in the retail store.” “the capturing device may include a handheld device (e.g., a smartphone, a tablet, a mobile station, a personal digital assistant, a laptop, and more) or a wearable device (e.g., smart glasses, a smartwatch, a clip-on camera).” ¶0132, “a positioning sensor may also be integrated with, or connected to, capturing device 125.” ¶0241, “the at least one image may be analyzed to determine a position (e.g., such as location, orientation, etc.) in the physical retail store to present the virtual visual indicator at. A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator at the determined position. In one example, the digital signal may include an indication of the position (e.g., in a coordinate system relative to at least one of the world, the retail store, one or more objects within the retail store, or the extended reality appliance (for example, at a particular point in time).”);
obtaining an associated first pose of the first mobile electronic device at a moment of capturing the first image (¶0241, “the at least one image may be analyzed to determine a position (e.g., such as location, orientation, etc.) in the physical retail store to present the virtual visual indicator at.” “machine learning model may be trained using training examples to determine information associated with visual indicators (e.g., such as position information, visual indicator characteristic information, etc.) from images and/or videos. An example of such training example may include a sample image and/or a sample video, together with a label indicating the desired determination of information for the sample image and/or the sample video. ¶0414, “analyzing the acquired image data to determine the positions in the retail store from which each of the detected plurality of online digital activities originated. The image data may be acquired by one or more cameras positioned in the retail store, each associated with a unique identifier that associates each image with a position in the retail store.” ¶0418, “the position information may further indicate an orientation of the one or more mobile devices during generation of one or more of the plurality of online digital activities. For example, one or more of the mobile devices may be provided with one or more of an inertial measuring unit (IMU), e.g., including a gyroscope, compass, or accelerometer for determining the orientation and position of the mobile device. The orientation of the mobile device may facilitate in determining the position of each mobile device with respect to the condition.”);
inputting the first image to a machine learning model trained to output an action based on context in an image and a location in the image associated with the action, thereby obtaining i) an action to be performed and ii) a location within the first image associated with the action (¶0108, “trained machine learning algorithms (also referred to as machine learning models and trained machine learning models in the present disclosure) may be used to analyze inputs and generate outputs” “a trained machine learning algorithm may include a classification algorithm, the input may include an image, and the inferred output may include a classification of an item depicted in the image. In yet another example, a trained machine learning algorithm may include a regression model, the input may include an image, and the inferred output may include an inferred value corresponding to an item depicted in the image (such as an estimated property of the item, such as size, volume, age of a person depicted in the image, distance from an item depicted in the image, and so forth). In an additional example, a trained machine learning algorithm may include an image segmentation model, the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, a trained machine learning algorithm may include an object detector, the input may include an image, and the inferred output may include one or more detected objects in the image and/or one or more locations of objects within the image.” ¶0128, “inventory data 246 that may be used to determine if additional products should be ordered from suppliers 115; employee data 248 (e.g., attendance data, records of training provided, evaluation and other performance-related communications, productivity information, etc.) that may be used to assign specific store associates to certain tasks”, Fig. 13 A. ¶0241, “A processor of server 135 may execute one or more operating instructions located in a non-transitory computer readable medium to analyze the at least one image to cause an extended reality appliance to present to a shopper a visual indicator associated with the alternative product in the particular retail store. For example, the at least one image may be analyzed to determine a position (e.g., such as location, orientation, etc.) in the physical retail store to present the virtual visual indicator at. A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator at the determined position. In one example, the digital signal may include an indication of the position (e.g., in a coordinate system relative to at least one of the world, the retail store, one or more objects within the retail store, or the extended reality appliance (for example, at a particular point in time). In another example, the at least one image may be analyzed to select of the visual indicator or of a characteristic of the visual indicator (e.g., such as size, color scheme, opacity, graphical content, textual content, and so forth). A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator with the selected characteristic.” ¶0515, “Depending on the information determined from the captured image data, additional actions or notifications may be generated to rectify or correct the store's inventory of wheat bread. This action or information may cause anything from a notification to a retail store associate about the anomalous condition that led to the anomalous transaction to an automated response, such as deployment of a robot or drone to restock shelves. The disclosed systems and methods, described in further detail below, seeks to seamlessly and accurately manage the inventory of a retail store and address anomalous transactions.”);
for the action to be performed, determining a position of the action to be performed within the virtual representation of the real space as an intersection between a known structure of the virtual representation of the real space and (¶0241, “A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator at the determined position. In one example, the digital signal may include an indication of the position (e.g., in a coordinate system relative to at least one of the world, the retail store, one or more objects within the retail store, or the extended reality appliance (for example, at a particular point in time). In another example, the at least one image may be analyzed to select of the visual indicator or of a characteristic of the visual indicator (e.g., such as size, color scheme, opacity, graphical content, textual content, and so forth). A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator with the selected characteristic.” ¶0519, “analyzing the image data may determine a possible cause of the anomalous transaction: wheat bread 3307 is out of stock while white bread 3305 is currently in stock. Based on the result of this analysis of the image data, system 3303 may generate and cause transmission of, for example, a notification to a handheld device 3309 used by a retail store associate or other person associated with the retail store. The notification may include information regarding the anomalous transaction and a likely cause of the anomalous transaction (e.g., white bread 3305 is currently in stock on aisle 4 and wheat bread is currently out of stock in aisle 4). Additionally or alternatively to providing the notification, system 3303 may additionally initiate actions that are manual or automated, e.g., activating a robot or drone as demonstrated in FIG. 4A, FIG. 4B, and FIG. 4C.”); and
generating a task to be performed, the task comprising the action to be performed and its position within the virtual representation of the real space (Fig. 13A, ¶0241, “A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator at the determined position. In one example, the digital signal may include an indication of the position (e.g., in a coordinate system relative to at least one of the world, the retail store, one or more objects within the retail store, or the extended reality appliance (for example, at a particular point in time). In another example, the at least one image may be analyzed to select of the visual indicator or of a characteristic of the visual indicator (e.g., such as size, color scheme, opacity, graphical content, textual content, and so forth). A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator with the selected characteristic.” ¶0520,” the alert or notification may show similar information as the information provided at 3401 such as details regarding the anomalous transaction (e.g., product, product type, brand, price, etc.) and a location (e.g., a store aisle) associated with the purchased product, but may differ in display, e.g., to demonstrate that the system or a store associate is in the process of taking an action, thus making the resolution of the anomalous transaction pending.” ¶0546, “the indicator of the at least one condition may be configured to prompt initiation of an action to address the at least one condition. The action may include a customer associate restocking the product type or ordering more of a particular product or product type from a supplier in order to correct the deficiency that causes the at least one condition.”).
Bronicki does not explicitly disclose a raycast from the first pose against a screen space coordinate of the location.
However, raycast technique is obvious in extended reality or augmented reality to pin point the location.
Powderly teaches a raycast from the first pose against a screen space coordinate of the location (Powderly, ¶0125, “The cone castings can involve casting thin, pencil rays with substantially little transverse width or casting rays with substantial transverse width (e.g., cones or frustums) from an AR display (of the wearable system) toward physical or virtual objects. Cone casting with a single ray may also be referred to as ray casting. Detailed examples of cone casting techniques are described in U.S. application Ser. No. 15/473,444, titled “Interactions with 3D Virtual Objects Using Poses and Multiple-DOF Controllers”, filed Mar. 29, 2017, the disclosure of which is hereby incorporated by reference in its entirety.” ¶0195, “Eye gaze or head pose may then be considered a source point for a cone cast or ray cast for virtual object selection.” ¶0272, “determine intersections of virtual objects in the user's environment with a head pose based raycast (or cone cast), determine the user's hand position, determine planar surface mesh or environment planar mesh (e.g., a mesh associated with a wall or a table), etc”).
Bronicki and Powderly are considered to be analogous art because all pertain to augmented reality. It would have been obvious before the effective filing date of the claimed invention to have modified Bronicki with the features of “a raycast from the first pose against a screen space coordinate of the location” as taught by Powderly. The suggestion/motivation would have been in order to determine intersections of virtual objects in the user's environment with a head pose based raycast (Powderly, ¶0272).
As to claim 2, claim 1 is incorporated and the combination of Bronicki and Powderly discloses sending the task to a second mobile electronic device being calibrated in the virtual representation of the real space such that the second mobile device has a known pose in the virtual representation of the real space, and displaying the task at a display of the second mobile electronic device as an AR-object within the virtual representation of the real space, the AR-object being located at the position of the action to be performed (Bronicki, Fig. 13A, ¶0241, “A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator at the determined position. In one example, the digital signal may include an indication of the position (e.g., in a coordinate system relative to at least one of the world, the retail store, one or more objects within the retail store, or the extended reality appliance (for example, at a particular point in time). In another example, the at least one image may be analyzed to select of the visual indicator or of a characteristic of the visual indicator (e.g., such as size, color scheme, opacity, graphical content, textual content, and so forth). A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator with the selected characteristic.” ¶0520,” the alert or notification may show similar information as the information provided at 3401 such as details regarding the anomalous transaction (e.g., product, product type, brand, price, etc.) and a location (e.g., a store aisle) associated with the purchased product, but may differ in display, e.g., to demonstrate that the system or a store associate is in the process of taking an action, thus making the resolution of the anomalous transaction pending.” ¶0546, “the indicator of the at least one condition may be configured to prompt initiation of an action to address the at least one condition. The action may include a customer associate restocking the product type or ordering more of a particular product or product type from a supplier in order to correct the deficiency that causes the at least one condition.”).
As to claim 3, claim 2 is incorporated and the combination of Bronicki and Powderly discloses guiding a user of the second mobile electronic device to the position of the action to be performed (Bronicki, Fig. 13A, ¶0241, “A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator at the determined position. In one example, the digital signal may include an indication of the position (e.g., in a coordinate system relative to at least one of the world, the retail store, one or more objects within the retail store, or the extended reality appliance (for example, at a particular point in time). In another example, the at least one image may be analyzed to select of the visual indicator or of a characteristic of the visual indicator (e.g., such as size, color scheme, opacity, graphical content, textual content, and so forth). A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator with the selected characteristic.” ¶0520,” the alert or notification may show similar information as the information provided at 3401 such as details regarding the anomalous transaction (e.g., product, product type, brand, price, etc.) and a location (e.g., a store aisle) associated with the purchased product, but may differ in display, e.g., to demonstrate that the system or a store associate is in the process of taking an action, thus making the resolution of the anomalous transaction pending.” ¶0546, “the indicator of the at least one condition may be configured to prompt initiation of an action to address the at least one condition. The action may include a customer associate restocking the product type or ordering more of a particular product or product type from a supplier in order to correct the deficiency that causes the at least one condition.”).
As to claim 4, claim 1 is incorporated and the combination of Bronicki and Powderly discloses obtaining an action to be performed comprises selecting the action to be performed from a set of candidate actions outputted from the machine learning model having the first image as input (Bronicki, ¶0108, “trained machine learning algorithms (also referred to as machine learning models and trained machine learning models in the present disclosure) may be used to analyze inputs and generate outputs” “a trained machine learning algorithm may include a classification algorithm, the input may include an image, and the inferred output may include a classification of an item depicted in the image. In yet another example, a trained machine learning algorithm may include a regression model, the input may include an image, and the inferred output may include an inferred value corresponding to an item depicted in the image (such as an estimated property of the item, such as size, volume, age of a person depicted in the image, distance from an item depicted in the image, and so forth). In an additional example, a trained machine learning algorithm may include an image segmentation model, the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, a trained machine learning algorithm may include an object detector, the input may include an image, and the inferred output may include one or more detected objects in the image and/or one or more locations of objects within the image.” ¶0128, “inventory data 246 that may be used to determine if additional products should be ordered from suppliers 115; employee data 248 (e.g., attendance data, records of training provided, evaluation and other performance-related communications, productivity information, etc.) that may be used to assign specific store associates to certain tasks”, Fig. 13 A. ¶0241, “A processor of server 135 may execute one or more operating instructions located in a non-transitory computer readable medium to analyze the at least one image to cause an extended reality appliance to present to a shopper a visual indicator associated with the alternative product in the particular retail store. For example, the at least one image may be analyzed to determine a position (e.g., such as location, orientation, etc.) in the physical retail store to present the virtual visual indicator at. A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator at the determined position. In one example, the digital signal may include an indication of the position (e.g., in a coordinate system relative to at least one of the world, the retail store, one or more objects within the retail store, or the extended reality appliance (for example, at a particular point in time). In another example, the at least one image may be analyzed to select of the visual indicator or of a characteristic of the visual indicator (e.g., such as size, color scheme, opacity, graphical content, textual content, and so forth). A digital signal may be provided to the extended reality appliance (e.g., through a digital communication device) to cause the extended reality appliance to present the visual indicator with the selected characteristic.” ¶0515, “Depending on the information determined from the captured image data, additional actions or notifications may be generated to rectify or correct the store's inventory of wheat bread. This action or information may cause anything from a notification to a retail store associate about the anomalous condition that led to the anomalous transaction to an automated response, such as deployment of a robot or drone to restock shelves. The disclosed systems and methods, described in further detail below, seeks to seamlessly and accurately manage the inventory of a retail store and address anomalous transactions.”).
As to claim 5, claim 4 is incorporated and the combination of Bronicki and Powderly discloses in the selecting comprises prompting a user to select the action to be performed from the set of candidate actions (Bronicki, ¶0023, “presenting the synthetic visual representation on a display; receiving from a user an input indicative of a selection of at least a portion of the synthetic visual representation; and in response to receiving the input, presenting on the display an actual image corresponding to the selected at least a portion of the synthetic visual representation, wherein the actual image was acquired with an image sensor.” ¶0101, “an establishment offering products for sale by direct selection by customers physically or virtually shopping within the establishment.” ¶0225, “By selecting each of the GUI features, the online customer may virtually jump to different locations or departments in retail store 105.” ¶0228, “it may improve customer satisfaction by enabling the shopper to select more suitable replacement items. Therefore, suggesting alternative items for out-of-stock items in a natural and efficient way is desired.” ¶0360, “the first information received from the customer may include a selection of at least one of a plurality of options presented to the customer.” ¶0384, “the selected action may be to query the customer, “Did you select a gallon of milk?” The selected action may be less specific if the sensed data reveals fewer details. For example, if all that is known about the container is the shape (e.g., the container may include milk, water, or soda), the selected action may be to ask the customer to pick what product the customer selected from a list of drink categories. In some embodiments, more than one selected action may be needed to refine what the second property is if the sensed data is limited. For example, a first action may include issuing a first query whether the customer selected a beverage and, if so, a second action may include issuing a second query to ask if the customer selected one of the beverages on a list.” ¶0441, “the at least one remedial action may selected based on sales data associated with the particular product and the retail store.” ¶0442-0445, “a prompt to restock a product display shelf in the retail store.” ¶0478-0479.).
As to claim 6, claim 1 is incorporated and the combination of Bronicki and Powderly discloses the virtual representation of the real space is represented in a 3D space (Bronicki, ¶0234, “an extended reality appliance may display 3D holograms overlaid on the real world where the user is located to render a mixed reality experience to the user.” ¶0493, “3D shape data received as part of camera location information or status information”).
As to claim 7, claim 1 is incorporated and the combination of Bronicki and Powderly discloses the virtual representation of the real space is represented in a 2D space (Bronicki, Fig. 11B, 11D, 13A-C, Fig. 15A, Fig.28A).
As to claim 8, claim 1 is incorporated and the combination of Bronicki and Powderly discloses the virtual representation of the real space is a virtual representation of a store, the virtual representation of the store comprising a number of known structures for displaying products/articles in the store (Bronicki, Fig. 11B, 11D, 13A-C, Fig. 15A, Fig.28A, ¶0234, “an extended reality appliance may display 3D holograms overlaid on the real world where the user is located to render a mixed reality experience to the user.”).
As to claim 9, the combination of Bronicki and Powderly discloses a non-transitory computer-readable storage medium having stored thereon instructions for implementing the method according to claim 1, when executed on one or more devices having processing capabilities (See claim 1 for detailed analysis.).
As to claim 10, the combination of Bronicki and Powderly discloses a server comprising circuitry configured to execute (Bronicki, ¶0113): an image obtaining function configured to obtain a first image and a first pose, of a first mobile electronic device at a moment the first mobile electronic device captured the first image, wherein the first mobile electronic device is calibrated in a virtual representation of a real space such that the first mobile electronic device has a known pose in the virtual representation of the real space; an action obtaining function configured to obtain i) an action to be performed and ii) a location within an image associated with the action to be performed by inputting the first image to a machine learning model trained to output an action based on context in an image and a location associated with the action in the image; a position determining function configured to, for the action to be performed, determine a position of the action to be performed within the virtual representation of the real space as an intersection between a known structure of the virtual representation of the real space and a raycast from the first pose against a screen space coordinate of the location associated with the action to be performed in the first image; and a task generating function configured to generate a task to be performed, the task comprising the action to be performed and its position within the virtual representation of the real space (See claim 1 for detailed analysis.).
As to claim 11, claim 10 is incorporated and the combination of Bronicki and Powderly discloses the server is implemented as a cloud server (Bronicki, ¶0113, ¶0115-0116.).
As to claim 12, claim 10 is incorporated and the combination of Bronicki and Powderly discloses a memory comprising information pertaining to the virtual representation of the space (Bronicki, ¶0118, ¶0127-0128, ¶0225, ¶0241).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bronicki et al. (US Pub 2023/0281549) discloses to provide a dynamic solution that will automatically monitor retail spaces and determine whether a disparity exists between a desired product placement and an actual product placement.
Florenzano et al. (US Pub 2023/0377027 A1) discloses virtual content may be rendered at a particular position within a 3D spatial map and overlaid onto a view of the real-world space.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YU CHEN whose telephone number is (571)270-7951. The examiner can normally be reached on M-F 8-5 PST Mid-day flex.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YU CHEN/Primary Examiner, Art Unit 2613
Prosecution Insights