DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This is in response to applicant’s amendment/response filed on 01/22/2026, which has been entered and made of record. Claims 1, 7, 17, and 22 have been amended. Claims 6, 10, and 21 have been cancelled. Claim 29 has been added. Claims 1-20 are pending in the application.
Response to Arguments
Applicant's arguments filed on 01/22/3036 have been fully considered but they are not persuasive or are rendered moot in view of the new grounds of rejection presented below (as necessitated by the amendment to claims 1 and 17).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2021/0192162 to Rodriguez et al. in view of U.S. PGPubs 2018/0181907 to Esmailzadeh, further in view of U.S. PGPubs 2017/0161674 to Jones et al., further in view of U.S. PGPubs 2019/0180485 to Kim et al..
Regarding claim 1, Rodriguez et al. teach a method of item recognition using augmented reality (par 0100, par 0408, par 0424), the method comprising:
capturing a plurality of images with an augmented reality (AR) device, the plurality of images including an item (par 0408, “The animation can be presented as an augmented reality overlay on the display of the person's portable device”, par 0443, “ A point of sale system 371 includes an item recognition portion coupled to a sensor 372 (e.g., one or more cameras, etc.). A display panel 373 has an item 374 resting on it, which is recognized and determined by the POS station to have a price of $2.49 (e.g., by reference to a database system)”);
extracting with the AR device image features from the plurality of images (par 0445, “From the sensor data, a class of the produce is recognized (e.g., by object recognition based on imagery, or chemical signature). The system may recognize, for example, that the bag contains apples. Based on the class, the system presents a listing of only the items in that class”);
extracting with the AR device a plurality of words on the item from the plurality of images (par 0102, “The object-identifying information can be a machine-readable identifier, such as a barcode or a steganographic digital watermark, either of which can convey a plural-bit payload. This information can additionally or alternatively comprise text—recognized by an optical character recognition engine”, par 0150, “Such a GPU can likewise be employed to perform a watermark or barcode decoding operation, or a fingerprint extraction operation, or an OCR operation, on multiple sets of data (e.g., the differently-transformed image sets) simultaneously.”, par 0169, “This image excerpt is passed to a text detector module that identifies at least one prominent alphabetic character. (Known OCR techniques can be used.)”);
identifying the item based on at least some of the image features and at least some of the plurality of words (par 0121, “an extraction process is applied to generate identification data corresponding to the item. The preferred embodiment applies a digital watermark decoding algorithm, but other identification technologies (e.g., barcode decoding, image fingerprinting, OCR, etc.) alternatively can be used”, , par 0316, “ An initial model of a total 3D volume presented for checkout can be generated based on sensed data (e.g., imagery). When an object in the pile is identified (e.g., by product markings, such as by fingerprinting, barcode, text OCR, or through use of other evidence), the object's shape (volume) is obtained from reference data stored in a database”, par 0250, “by color histogram analysis, the system may make a tentative identification of an item as, e.g., a six-pack of Coke. With this tentative identification, the system can obtain—from the database—information about the configuration of such product, and can use this information to discern the pose or orientation of the product as depicted in the camera imagery”, par 0454, “The product code identifies the product (e.g., Lebanon Bologna), and allows the system to recall the price for that item (e.g., $4.39/pound)”);
accessing with the AR device at least one database of information relating to the item (par 0102, “. This information can additionally or alternatively comprise text—recognized by an optical character recognition engine. Still further, the product can be identified by other markings, such as by image fingerprint information that is matched to reference fingerprint information in a product database”, par 0316, “ An initial model of a total 3D volume presented for checkout can be generated based on sensed data (e.g., imagery). When an object in the pile is identified (e.g., by product markings, such as by fingerprinting, barcode, text OCR, or through use of other evidence), the object's shape (volume) is obtained from reference data stored in a database”, par 0314-0316, “this weight can be analyzed and modeled in terms of component weights from individual objects—using reference weight data for such objects retrieved from a database …. When an object in the pile is identified (e.g., by product markings, such as by fingerprinting, barcode, text OCR, or through use of other evidence), the object's shape (volume) is obtained from reference data stored in a database”, par 0455-0456, “The scale includes a camera that captures an image of the package, and extracts the wrapper medium identifier from such imagery. The scale sends the extracted medium identifier—together with the other product details (e.g., product code, product name, measured weight, price per pound, total price) to the store's central computer database for storage”).
But Rodriguez et al. keep silent for teaching determining to deliver the item based on the information relating to the item and confirmation of delivery of the item.
In related endeavor, Esmailzadeh teach determining to deliver the item based on the information relating to the item (par 0039, “ Inventory verifications 224 may be confirmations of what particular inventory item is possessed by a particular mobile delivery vehicle that corresponds to a particular mobile delivery device 110 of the one or more mobile delivery devices 110 “, par 0048, “Transaction verification files 236 may correspond to files that evidence a completed sale and delivery of an item in inventory. Transaction verification files 236 may include credit card payment verification, barcode out scanning of the item of inventory, confirmation from the user device 108, confirmation from a mobile delivery device 110, audio recordings of the transaction, video or photographic recordings of a transaction, fingerprint confirmation of a transaction, eye recognition confirmation of a transaction, and other methods”, par 0063-0066, “he regional inventory lists 232 may be filtered at block 408 to remove any inventory listing that is not responsive to the type of product determined by the user attributes 218. For instance, the user attributes 218 may indicate that the user device 108 desires a regional inventory lists 232 that is populated by inventory of a particular category (i.e. chocolate candies, soft drinks, sandwiches, cigarettes, hot drinks, snacks, etc.). The regional inventory lists 232 may also be filtered at block 408 by what particular inventory a mobile delivery device 110 possess. This would then result in displaying a regional inventory list that indicated which mobile delivery devices 110 possess all the inventory for which the user device 108 is interested in. This method could be used to incentivize a user device 108 to purchase inventories owned by a single mobile delivery device 110 ….. The inventory purchase request will indicate to the mobile delivery device 110 that the user device 108 making the inventory purchase request would like to engage in a purchase and delivery transaction with the mobile delivery device 110. The mobile delivery device 110 would then take action to facilitate the delivery and sale of the item of inventory”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. to include determining to deliver the item based on the information relating to the item and confirmation of delivery of the item as taught by Esmailzadeh to allow remote users to order goods from any number of mobile delivery vehicles to make activities in the day to day lives of users more convenient, secure, and efficient.
But Rodriguez et al. as modified by Esmailzadeh keep silent for teaching updating an inventory database upon confirmation of delivery of the item.
In related endeavor, Jones et al. teach updating an inventory database upon confirmation of delivery of the item (par 0003, “Inventory management systems of such retail sales facilities generally update the perpetual inventory in their inventory management database based on the time of delivery. Such updates are made on an assumption that the products are delivered within the scheduled time window and that the products are binned and worked to the sales floor immediately or shortly after the products are delivered”, par 0036, “FIG. 4 shows one embodiment of an operational workflow in implementing a method 400 in order to determine whether to update the electronic data in the inventory management database 140 representing the perpetual inventory of products 190 at the retail sales facility 110 and if so, a delay period determined based on the processed factors associated with the products 190 “, par 0039, “If the electronic data retrieved from the inventory management database 140 indicates that the picks of the cold chain product 190 off the delivery truck were complete, the control unit 210 is programmed to send a signal to the inventory management database to update the perpetual inventory (step 420) “, par 0044-0045, “If the electronic data retrieved from the inventory management database 140 indicates that a product 190 was delivered outside of the projected time for delivery, the control unit 210 is programmed to send a signal to the inventory management database 140 to delay the updating of the perpetual inventory or to update the perpetual inventory (step 420) as appropriate”)
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh to include updating an inventory database upon confirmation of delivery of the item as taught by Jones et al. to update perpetual inventory at a retail sales facility after products are delivered to a retail sales facility to provide precisely and efficiently inventory management to reduce the costs associated with product sorting at the retail sales facility and increase the productivity of the workers at the retail sales facility.
But Rodriguez et al. as modified by Esmailzadeh and Jones et al. do not explicitly teach receiving with an augmented reality (AR) device an item identification for an item that is to be retrieved from a storage device; outputting a location of the item within the storage device using at least one output device of the AR device; after outputting the location, capturing a plurality of images with the AR device.
In related endeavor, Kim et al. teach receiving with an augmented reality (AR) device an item identification for an item that is to be retrieved from a storage device; outputting a location of the item within the storage device using at least one output device of the AR device (par 0330-0332, “Using the specific information 1000, the processor 870 acquires a current location (GPS) of the vehicle, a visual traveling record (a visual odometry), a location of a destination (a point-of-interest (POI) GPS), a camera-captured image, a preference level of a POI in the vicinity of the vehicle, and meta data on a destination or the POI”); after outputting the location, capturing a plurality of images with the AR device, the plurality of images including an item (par 0340-0342, “The AR graphic rendering means outputting the graphic object on the windshield and the window so the graphic object is superimposed on the destination when the driver takes a look at the destination. the camera-captured image, the coordinates of the graphic object for the destination in the camera coordinate system, and the information relating to the destination, the processor 870 (or the AR graphic rendering engine) outputs destination (POI) information so the destination (POI) information is superimposed on the destination (outputs the destination (POI) information to the vicinity of the destination) (S1080).”, Figs 11(a)-11(b), par 0349, “With reference to FIG. 11B(a), the processor 870 identifies a preset destination 1100 from an image that is received through the camera. For example, the processor 870 identifies an object that corresponds to the preset image from the image”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh and Jones et al. to include receiving with an augmented reality (AR) device an item identification for an item that is to be retrieved from a storage device; outputting a location of the item within the storage device using at least one output device of the AR device; after outputting the location, capturing a plurality of images with the AR device as taught by Kim et al. to cause an output position and a shape of the graphic object to be variable, based on a location and a shape of the destination changing based on a camera's view of the destination while traveling.
Regarding claim 4, Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. teach all the limitation of claim 1, and Rodriguez et al. further teach wherein the item comprises a consumable (par 0341, par 0379, food and so on).
Claim(s) 2-3, 5, 7-9, 11-14, and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2021/0192162 to Rodriguez et al. in view of U.S. PGPubs 2018/0181907 to Esmailzadeh, further in view of U.S. PGPubs 2017/0161674 to Jones et al., further in view of U.S. PGPubs 2019/0180485 to Kim et al., further in view of U.S. PGPubs 2023/0238098 to Langan et al..
Regarding claim 2, Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. teach all the limitation of claim 1, but do not explicitly teach wherein the item comprises a medication.
In related endeavor, Langan et al. teach wherein the item comprises a medication (Figs 3A-3B, par 0033, “AR glasses 130 is configured to facilitate an intake of medications into a stock room. AR glasses 130 are configured to overlay graphic information regarding inventory control over the real world environment perceived by the user”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. to include wherein the item comprises a medication as taught by Langan et al. to introduce augmented reality into medication area to identify medication, medical device, and user to improve systems and methods of hands-free medication tracking.
Regarding claim 3, Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. teach all the limitation of claim 1, but do not explicitly teach wherein the item comprises a medical device.
In related endeavor, Langan et al. teach wherein the item comprises a medical device (Figs 3A-3B, par 0033, “AR glasses 130 is configured to facilitate an intake of medications into a stock room. AR glasses 130 are configured to overlay graphic information regarding inventory control over the real world environment perceived by the user”, par 0027, “. The image data may be captured for specific items (e.g., medications, medical equipment), medication dispensers, medical devices (e.g., infusion pump), or physical structures within the clinical area (e.g., signs, floor tiles, doors, trashcans, or other physical landmarks)”, par 0039, “The AR glasses 130 may detect a medication dispensing cabinet or other medical device within the field of view”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. to include wherein the item comprises a medical device as taught by Langan et al. to introduce augmented reality into medication area to identify medication, medical device, and user to improve systems and methods of hands-free medication tracking.
Regarding claim 5, Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. teach all the limitation of claim 1, but do not explicitly teach wherein the item is stored in a medical storage area.
In related endeavor, Langan et al. teach wherein the item is stored in a medical storage area (par 0029, “Other AR projections may highlight a position of medications for retrieval, for example by providing a high contrast dot, circle, or other visual identifier. Yet other AR projections may identify a position for placement of the medication, such as by identifying a particular compartment or bin in a multi-tiered medication shelf”, Fig 3A, par 0034-0035, “Another example use case is medication drop-off to specific locations, such as bin arrays, medication shelves, and dispensing machines. The user action, or medication drop-off, may be determined based on the detected location (e.g. near a dispensing machine or medication shelf) or authenticated user (e.g. a pharmacist). First, the user may look at a label of the medication for identification, as previously shown in AR display 332A. AR message 310B may then be projected to the user as shown in AR display 332B, indicating to the user where to drop off the medication within medication shelf 314, or “Storeroom 120, Shelf B-1””, par 0039, “the AR glasses 130 may detect a medication dispensing cabinet or other medical device within the field of view”, par 0051, “if a clinician is detected to be standing in a medical supply closet”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. to include wherein the item is stored in a medical storage area as taught by Langan et al. to introduce augmented reality into medication area to identify medication, medical device, user, a medical storage area to improve systems and methods of hands-free medication tracking and management.
Regarding claim 7, Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. teach all the limitation of claim 1, but do not explicitly teach wherein the storage device comprises: a storage-device processor; and a storage-device memory, the storage device memory comprising an inventory database identifying: items stored in the storage device; and a location in the storage device of the items stored in the medical storage device.
In related endeavor, Langan et al. further teach wherein the storage device comprises: a storage-device processor (par 0063); and a storage-device memory, the storage device memory comprising an inventory database identifying (par 0017, “The lookup request may include information such as location, viewing angle, images, video, audio, user identifier, user role identifier, and user eye tracking information. Server 114 may query database 116 to identify medication 120 based on the lookup request ….A message indicating the user action may then be sent to server 114 to cause an update in an inventory stored in database 116.”): items stored in the storage device; and a location in the medical storage device of the items stored in the storage device (par 0025, “When the item is placed in a new location, processor 134 (or server 114) may automatically associate the item with the new location in the database, and create a record indicating that the user placed the item at the new location. In this regard, the record may include a geolocation or shelf or storage identifier, an identifier of the item, and an identifier of the user who placed the item at the new location”, par 0034, “User 110 wearing AR glasses 130 may simply direct gaze towards barcode 312A on a label affixed to the medication, as shown in AR display 332A. AR message 310A may then be projected to the user, indicating the medication is detected as Aspirin 650 mg, 200 count. A corresponding update message may be sent to server 114 to update an inventory of the stock room stored in database 116”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. to include wherein the storage device comprises: a storage-device processor; and a storage-device memory, the storage device memory comprising an inventory database identifying: items stored in the storage device; and a location in the storage device of the items stored in the medical storage device as taught by Langan et al. to introduce augmented reality into medication area to identify medication, medical device, user, a medical storage area to improve systems and methods of hands-free medication tracking and management.
Regarding claim 8, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., and Langan et al. teach all the limitation of claim 7, and Langan et al. further teach wherein the storage device comprises an automated dispensing cabinet (ADC) (par 0016, “User actions may include, for example, intake of medication from restock areas, retrieval of medication from automated dispensing machines, bins, or shelves, placement of medication into shelves or bins, administration of medication to a patient, and disposal of excess medications “, par 0018, par 0033, “intake of medication from restock areas, retrieval of medication from specific automated dispensing machines, bins, or shelves, placement or delivery of medication into specific automated dispensing machines, bins, or shelves, administration of a medication to a patient, and disposal of excess medications”). This would be obvious for the same reason given in the rejection for claim 6.
Regarding claim 9, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., and Langan et al. teach all the limitation of claim 8, and Langan et al. further teach wherein the ADC comprises a plurality of drawers, each of which drawers comprising a plurality of compartments, wherein at least some of the items are stored each in a unique compartment (par 0016, “User actions may include, for example, intake of medication from restock areas, retrieval of medication from automated dispensing machines, bins, or shelves, placement of medication into shelves or bins, administration of medication to a patient, and disposal of excess medications “, par 0018, “a user is accessing a matrix drawer including several different storage locations (e.g., bins or pockets) for different items “, par 0024, “ When the object is received into the user's hand, the algorithm may detect the placement and begin identification of the object through image recognition, or by analyzing captured images for identifying features or codes affixed to the object (e.g., barcode or Q-code)”, Fig 3A, par 0035, “Database 116 may include a virtual 3D model of medication shelf 314 that includes metadata specifying the preferred positions of particular medications within shelves A-1, B-1, A-2, and B-2”). This would be obvious for the same reason given in the rejection for claim 6.
Regarding claim 11, Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. teach all the limitation of claim 1, but do not explicitly teach wherein the at least one database of information relating to the item comprises a medication database stored in memory accessible by the AR device via a communication network.
In related endeavor, Langan et al. teach wherein the at least one database of information relating to the item comprises a medication database stored in memory accessible by the AR device via a communication network (par 0017, “User 110 may wear AR glasses 130 to track various medications, such as medication 120. To identify medication 120, sensors of AR glasses 130 may provide data for a lookup request to be sent to server 114 via network 112. The lookup request may include information such as location, viewing angle, images, video, audio, user identifier, user role identifier, and user eye tracking information. Server 114 may query database 116 to identify medication 120 based on the lookup request. Further, AR glasses 130 may determine and confirm a user action with respect to medication 120, such as adding, removing, administering, or disposing of medications. A message indicating the user action may then be sent to server 114 to cause an update in an inventory stored in database 116. “, par 0033, “FIG. 3A and FIG. 3B depict various example augmented reality displays of an augmented reality device, such as AR glasses 130, for hands-free medication tracking, according to various aspects of the subject technology. AR glasses 130 is configured to facilitate an intake of medications into a stock room. AR glasses 130 are configured to overlay graphic information regarding inventory control over the real world environment perceived by the user. AR glasses 130 are configured to communicate a location of the user to a server 114, including the user orientation within the current environment and current direction of sight. Visual information may be converted to digital information and sent to the server, which may then use object or image recognition to identify objects and their specific locations within a three-dimensional space within the view of the user. Based on the current location of the user, the system may identify actions that are required to be performed by the user regarding an inventory of medical items. Such actions may include, for example, intake of medication from restock areas, retrieval of medication from specific automated dispensing machines, bins, or shelves, placement or delivery of medication into specific automated dispensing machines, bins, or shelves, administration of a medication to a patient, and disposal of excess medications”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. to include wherein the at least one database of information relating to the item comprises a medication database stored in memory accessible by the AR device via a communication network as taught by Langan et al. to introduce augmented reality into medication area to identify medication, medical device, user, a medical storage area to improve systems and methods of hands-free medication tracking and management.
Regarding claim 12, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., and Langan et al. teach all the limitation of claim 11, and Langan et al. further teach wherein determining to deliver the item is further based on patient information, wherein the patient information is associated with a recipient of the item (par 0025, “When an item is picked up and/or moved, processor 134 (or server 114) may automatically identify the location from which the item is picked up, and associate an identification of the user wearing the glasses with the movement or placement of the item “, par 0033, “Based on the current location of the user, the system may identify actions that are required to be performed by the user regarding an inventory of medical items. Such actions may include, for example, intake of medication from restock areas, retrieval of medication from specific automated dispensing machines, bins, or shelves, placement or delivery of medication into specific automated dispensing machines, bins, or shelves, administration of a medication to a patient, and disposal of excess medications”, par 0048, “the determining may be based on the authenticated user (e.g. data from retinal scanner 156), a currently detected location (e.g. data from location sensor 158, RGB camera 152, time of flight camera 154), an intended location for the medication (e.g. data from RGB camera 152 and time of flight camera 154 matched to virtual 3D models in database 116), a patient identification (e.g. from RGB camera 152 or RFID reader 146), or other data from sensors 150 and/or communication interface 140 …. The 3D mapping may be generated by processing the data from sensors 150 through a machine learning model trained with model data for medication shelves, bin arrays, automatic dispensing machines, and other locations for retrieving and delivering medications. The 3D mapping may then be matched to virtual 3D models in database 116, wherein the virtual 3D models also specify the positions of particular medications, including preferred positions for drop off”). This would be obvious for the same reason given in the rejection for claim 11.
Regarding claim 13, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., and Langan et al. teach all the limitation of claim 12, and Langan et al. further teach further comprising retrieving the patient information from a patient database (Fig 3B, par 0041, “ the user may direct gaze at barcode 312C attached to a wrist of the patient to identify patient 318, as shown in AR display 332E. Alternative implementations may use RFID tags or other means of identifying patient 318. AR glasses 130 may then lookup a treatment regimen for patient 318 and provide administration instructions for the user in AR message 310E, which instructs the user to administer orally twice daily or as needed”, par 0070, “ The message flow of FIG. 6 shows messages exchanged between several entities which can be included in an augmented reality system including a hospital information server 690 (e.g., electronic medical record system, patient information system, laboratory information system, time and attendance system, security system, or the like) and a medical device 692 such as an infusion pump or medication dispensing cabinet”). This would be obvious for the same reason given in the rejection for claim 11.
Regarding claim 14, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., and Langan et al. teach all the limitation of claim 13, and Langan et al. further teach wherein the patient database is stored in memory accessible by the AR device via the communication network (par 0033-0034, “ Such actions may include, for example, intake of medication from restock areas, retrieval of medication from specific automated dispensing machines, bins, or shelves, placement or delivery of medication into specific automated dispensing machines, bins, or shelves, administration of a medication to a patient, and disposal of excess medications. The different modes or actions may be triggered based on optical character recognition of information such as a schedule or other electronic medical record. …. As another example, if the system determines that a user's shift is ended, the AR glasses 130 may adjust resources to a configuration that prioritizes passive data collection. The determination may be based on one or more communications between the AR glasses 130 and a hospital information system such as a time and attendance system …..fields such as authenticated clinician ID or user ID, patient ID, current location, and other data may also be written for each record, enabling tracking of item movement according to user, patient, location, or other criteria. Thus, user 110 merely needs to look at the medication before drop-off to perform an automatic inventory record update in database 116, which requires no handling of handheld scanners, updating of paperwork, or other burdensome manual processes”, par 0084, “As shown in FIG. 6, data is transmitted directly between entities. However, in some implementations, the data may be transmitted via an intermediate server or data store. In such instances, the communication may identify the availability of data and the location of the data (e.g., file name or record identifier)”). This would be obvious for the same reason given in the rejection for claim 11.
Regarding claim 29, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al. teach all the limitation of claim 1, but keep silent for teaching further comprising: outputting delivery information relating to the item to a user via the at least one output device of the AR device.
In related endeavor, Langan et al. teach further comprising: outputting delivery information relating to the item to a user via the at least one output device of the AR device (par 0057, “From the dispatch or approval location, a clinician or other medical worker may bring the filled medication orders to patients or wards for dispensing to the patient. These workers may also be wearing AR glasses. As they approach the dispatch location, the AR glasses may detect that the worker is preparing to make a delivery run through the facility. The AR glasses or other system actor may survey the items waiting delivery and identify a set that maximizes the delivery run by, for example, grouping orders for a similar ward together. This grouping may be presented as augmented reality content when the user looks into the dispatch location. The items to be included on the next run may be highlighted or listed. The AR glasses may also provide routing information as augmented reality content to guide the worker through the facility along an efficient route. An efficient route may be one that, for example, avoids areas being cleaned, avoids backtracking, minimizes elevator rides, etc.”)
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Kim et al. to include further comprising: outputting delivery information relating to the item to a user via the at least one output device of the AR device as taught by Langan et al. to provide routing information as augmented reality content to guide the worker through the facility along an efficient route (avoids areas being cleaned, avoids backtracking, minimizes elevator rides, etc.).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2021/0192162 to Rodriguez et al. in view of U.S. PGPubs 2018/0181907 to Esmailzadeh, further in view of U.S. PGPubs 2017/0161674 to Jones et al., further in view of U.S. PGPubs 2019/0180485 to Kim et al., further in view of U.S. PGPubs 2023/0238098 to Langan et al., further in view of U.S. Patent 11941679 to Manyam.
Regarding claim 15, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., and Langan et al. teach all the limitation of claim 14, but do not explicitly teach wherein determining to deliver the item based on the information relating to the item further comprises determining that the item is not expired.
In related endeavor, Manyam teaches wherein determining to deliver the item based on the information relating to the item further comprises determining that the item is not expired (Fig 2, col 6:13-35, “ the user 204 may interact with the portable device 205 by selecting (e.g., via a touch screen) a display control, such as the “More Info” control 215. If the user selects the “More Info” control 215, the portable device 205 may submit a request to the inventory management system 150 for additional item information”, col 9:9-39, “Upon identifying the picked item and the quantity, an item expiration date may be determined for the picked item, as in 412. If the quantity of the item picked is greater than one, each item may have a different expiration date that is determined. In some implementations, the expiration date may be included in the picked item and detected from the images captured of the picked item. For example, a visual identifier may be included on the item that identifies the expiration date. In other implementations, the inventory management system may maintain in an item expiration data store the expiration information for each item and, when the picked item is identified, the inventory management system may identify the expiration date from stored information”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., and Langan et al. to include wherein determining to deliver the item based on the information relating to the item further comprises determining that the item is not expired as taught by Manyam to receive a recommendation to pick a replacement item so they do not have to return to the materials handling facility early to replace the expired item.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2021/0192162 to Rodriguez et al. in view of U.S. PGPubs 2018/0181907 to Esmailzadeh, further in view of U.S. PGPubs 2017/0161674 to Jones et al., further in view of U.S. PGPubs 2019/0180485 to Kim et al., further in view of U.S. PGPubs 2023/0238098 to Langan et al., further in view of U.S. Patent 11941679 to Manyam, further in view of U.S. PGPubs 2007/0021964 to Maenishi et al..
Regarding claim 16, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Kim et al., Langan et al., and Manyam teach all the limitation of claim 15, but do not explicitly teach wherein determining to deliver the item based on the information relating to the item further comprises determining that a lot number of the item is not associated with a recall.
In related endeavor, Maenishi et al. teach wherein determining to deliver the item based on the information relating to the item further comprises determining that a lot number of the item is not associated with a recall (par 0008, “the information items stored in the IC tag of an appliance is collected and used for checking whether the current appliance is the appliance to be recalled so as to identify whether the current appliance is the appliance to be recalled. Therefore, unlike the conventional method, it is possible to automatically identify the locations of the appliances to be recalled without making any recall notification of appliances by using newspaper advertisement or the like”, par 0095, “The recall database apparatus 37 instructs the IC tag reader and writer 34 that is set at each house 32 to read the information items stored in the IC tag 48 of the television receiving set 42 via the Internet 36 (S2). The IC tag reader and writer 34 sends the read information items to the recall database apparatus 37. Therefore, the recall database apparatus 37 receives the information items (S4). The recall database apparatus 37 checks the received information items with the recall product information 96 stored in the database unit 94 (S6). In other words, the recall database apparatus 37 checks whether the television receiving set 42 corresponding to the received information is the product to be recalled”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Rodriguez et al. as modified by Esmailzadeh, Jones, Kim et al., Langan et al., and Manyam to include wherein determining to deliver the item based on the information relating to the item further comprises determining that a lot number of the item is not associated with a recall as taught by Maenishi et al. to provide good and quality product to customer.
Claim(s) 17-20 and 21-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2023/0238098 to Langan et al. in view of U.S. PGPubs 2021/0192162 to Rodriguez et al., further in view of U.S. PGPubs 2018/0181907 to Esmailzadeh, further in view of U.S. PGPubs 2017/0161674 to Jones et al..
PNG
media_image1.png
302
298
media_image1.png
Greyscale
Regarding claim 17, Langan et al. teach a system for item recognition using augmented reality (par 0005, “a wearable augmented reality device comprising: a display interface for presenting a graphical user interface including at least one opaque or semi-transparent graphic element; a communication interface; a location sensor for detecting location information identifying a location of the wearable augmented reality device; an environment sensor for capturing information within the location”), the system comprising: a storage device comprising: a storage-device processor (par 0063); and a storage-device memory comprising an inventory database identifying (Figs 1-2, par 0017, par 0019, ““The lookup request may include information such as location, viewing angle, images, video, audio, user identifier, user role identifier, and user eye tracking information. Server 114 may query database 116 to identify medication 120 based on the lookup request ….A message indicating the user action may then be sent to server 114 to cause an update in an inventory stored in database 116.”): items stored in the storage device; and a location in the medical storage device of the items stored in the medical storage device (par 0025, “When the item is placed in a new location, processor 134 (or server 114) may automatically associate the item with the new location in the database, and create a record indicating that the user placed the item at the new location. In this regard, the record may include a geolocation or shelf or storage identifier, an identifier of the item, and an identifier of the user who placed the item at the new location”, par 0034, “User 110 wearing AR glasses 130 may simply direct gaze towards barcode 312A on a label affixed to the medication, as shown in AR display 332A. AR message 310A may then be projected to the user, indicating the medication is detected as Aspirin 650 mg, 200 count. A corresponding update message may be sent to server 114 to update an inventory of the stock room stored in database 116”, par 0029, “Other AR projections may highlight a position of medications for retrieval, for example by providing a high contrast dot, circle, or other visual identifier. Yet other AR projections may identify a position for placement of the medication, such as by identifying a particular compartment or bin in a multi-tiered medication shelf”, par 0035, “Another example use case is medication drop-off to specific locations, such as bin arrays, medication shelves, and dispensing machines. The user action, or medication drop-off, may be determined based on the detected location (e.g. near a dispensing machine or medication shelf) or authenticated user (e.g. a pharmacist). First, the user may look at a label of the medication for identification, as previously shown in AR display 332A. AR message 310B may then be projected to the user as shown in AR display 332B, indicating to the user where to drop off the medication within medication shelf 314, or “Storeroom 120, Shelf B-1””, par 0039, “the AR glasses 130 may detect a medication dispensing cabinet or other medical device within the field of view”, par 0051, “if a clinician is detected to be standing in a medical supply closet”); and an augmented reality (AR) device communicatively coupled to the storage device, the AR device comprising: a camera; a display; a memory comprising computer executable instructions; and a processor configured to (Fig 2, par 0019, “FIG. 2 depicts an example block diagram of an augmented reality device, or AR glasses 130, for hands-free medication tracking, according to various aspects of the subject technology. AR glasses 130 includes data bus 132, processor 134, memory 136, communication interface 140, sensors 150, display interface 164, identity access management (IAM) interface 168, audio interface 170, power source 180, and power harvester 182. Memory 136 includes non-volatile data store 137”):
receive an item identification for an item that is to be retrieved from within the storage device; output a location of the item within the storage device using the at least one output device (Fig 3A, par 0033-0035, “ AR glasses 130 is configured to facilitate an intake of medications into a stock room …. Based on the current location of the user, the system may identify actions that are required to be performed by the user regarding an inventory of medical items. Such actions may include, for example, intake of medication from restock areas, retrieval of medication from specific automated dispensing machines, bins, or shelves, placement or delivery of medication into specific automated dispensing machines, bins, or shelves, administration of a medication to a patient, and disposal of excess medications …the user may look at a label of the medication for identification, as previously shown in AR display 332A. AR message 310B may then be projected to the user as shown in AR display 332B, indicating to the user where to drop off the medication within medication shelf 314, or “Storeroom 120, Shelf B-1”. For example, the data from RGB camera 152 and time of flight camera 154 may be used to build a 3D mapping of the surrounding environment. Database 116 may include a virtual 3D model of medication shelf 314 that includes metadata specifying the preferred positions of particular medications within shelves A-1, B-1, A-2, and B-2. By matching the 3D mapping to the virtual 3D model, the preferred positions for dropping off the identified medication may be determined to position AR indicator 316 and AR message 310B within an AR projection output by display interface 164”).
But Langan et al. keep silent for teaching capturing a plurality of images with an augmented reality (AR) device, the plurality of images including an item; extracting with the AR device image features from the plurality of images; extracting with the AR device a plurality of words on the item from the plurality of images; identifying the item based on at least some of the image features and at least some of the plurality of words; accessing with the AR device at least one database of information relating to the item.
In related endeavor, Rodriguez et al. teach capturing a plurality of images with an augmented reality (AR) device, the plurality of images including an item (par 0408, “The animation can be presented as an augmented reality overlay on the display of the person's portable device”, par 0443, “ A point of sale system 371 includes an item recognition portion coupled to a sensor 372 (e.g., one or more cameras, etc.). A display panel 373 has an item 374 resting on it, which is recognized and determined by the POS station to have a price of $2.49 (e.g., by reference to a database system)”);
extracting with the AR device image features from the plurality of images (par 0445, “From the sensor data, a class of the produce is recognized (e.g., by object recognition based on imagery, or chemical signature). The system may recognize, for example, that the bag contains apples. Based on the class, the system presents a listing of only the items in that class”);
extracting with the AR device a plurality of words on the item from the plurality of images (par 0102, “The object-identifying information can be a machine-readable identifier, such as a barcode or a steganographic digital watermark, either of which can convey a plural-bit payload. This information can additionally or alternatively comprise text—recognized by an optical character recognition engine”, par 0150, “Such a GPU can likewise be employed to perform a watermark or barcode decoding operation, or a fingerprint extraction operation, or an OCR operation, on multiple sets of data (e.g., the differently-transformed image sets) simultaneously.”, par 0169, “This image excerpt is passed to a text detector module that identifies at least one prominent alphabetic character. (Known OCR techniques can be used.)”);
identifying the item based on at least some of the image features and at least some of the plurality of words (par 0121, “an extraction process is applied to generate identification data corresponding to the item. The preferred embodiment applies a digital watermark decoding algorithm, but other identification technologies (e.g., barcode decoding, image fingerprinting, OCR, etc.) alternatively can be used”, , par 0316, “ An initial model of a total 3D volume presented for checkout can be generated based on sensed data (e.g., imagery). When an object in the pile is identified (e.g., by product markings, such as by fingerprinting, barcode, text OCR, or through use of other evidence), the object's shape (volume) is obtained from reference data stored in a database”, par 0250, “by color histogram analysis, the system may make a tentative identification of an item as, e.g., a six-pack of Coke. With this tentative identification, the system can obtain—from the database—information about the configuration of such product, and can use this information to discern the pose or orientation of the product as depicted in the camera imagery”, par 0454, “The product code identifies the product (e.g., Lebanon Bologna), and allows the system to recall the price for that item (e.g., $4.39/pound)”);
accessing with the AR device at least one database of information relating to the item (par 0102, “. This information can additionally or alternatively comprise text—recognized by an optical character recognition engine. Still further, the product can be identified by other markings, such as by image fingerprint information that is matched to reference fingerprint information in a product database”, par 0316, “ An initial model of a total 3D volume presented for checkout can be generated based on sensed data (e.g., imagery). When an object in the pile is identified (e.g., by product markings, such as by fingerprinting, barcode, text OCR, or through use of other evidence), the object's shape (volume) is obtained from reference data stored in a database”, par 0314-0316, “this weight can be analyzed and modeled in terms of component weights from individual objects—using reference weight data for such objects retrieved from a database …. When an object in the pile is identified (e.g., by product markings, such as by fingerprinting, barcode, text OCR, or through use of other evidence), the object's shape (volume) is obtained from reference data stored in a database”, par 0455-0456, “The scale includes a camera that captures an image of the package, and extracts the wrapper medium identifier from such imagery. The scale sends the extracted medium identifier—together with the other product details (e.g., product code, product name, measured weight, price per pound, total price) to the store's central computer database for storage”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Langan et al. as modified by Rodriguez et al. to include capturing a plurality of images with an augmented reality (AR) device, the plurality of images including an item; extracting with the AR device image features from the plurality of images; extracting with the AR device a plurality of words on the item from the plurality of images; identifying the item based on at least some of the image features and at least some of the plurality of words; accessing with the AR device at least one database of information relating to the item as taught by Rodriguez et al. to analyzed for product recognition purposes through captured images to make activities in the day to day lives of users more convenient, secure, and efficient.
But Langan et al. as modified by Rodriguez et al. keep silent for teaching determining to deliver the item based on the information relating to the item and confirmation of delivery of the item.
In related endeavor, Esmailzadeh teach determining to deliver the item based on the information relating to the item (par 0039, “ Inventory verifications 224 may be confirmations of what particular inventory item is possessed by a particular mobile delivery vehicle that corresponds to a particular mobile delivery device 110 of the one or more mobile delivery devices 110 “, par 0048, “Transaction verification files 236 may correspond to files that evidence a completed sale and delivery of an item in inventory. Transaction verification files 236 may include credit card payment verification, barcode out scanning of the item of inventory, confirmation from the user device 108, confirmation from a mobile delivery device 110, audio recordings of the transaction, video or photographic recordings of a transaction, fingerprint confirmation of a transaction, eye recognition confirmation of a transaction, and other methods”, par 0063-0066, “he regional inventory lists 232 may be filtered at block 408 to remove any inventory listing that is not responsive to the type of product determined by the user attributes 218. For instance, the user attributes 218 may indicate that the user device 108 desires a regional inventory lists 232 that is populated by inventory of a particular category (i.e. chocolate candies, soft drinks, sandwiches, cigarettes, hot drinks, snacks, etc.). The regional inventory lists 232 may also be filtered at block 408 by what particular inventory a mobile delivery device 110 possess. This would then result in displaying a regional inventory list that indicated which mobile delivery devices 110 possess all the inventory for which the user device 108 is interested in. This method could be used to incentivize a user device 108 to purchase inventories owned by a single mobile delivery device 110 ….. The inventory purchase request will indicate to the mobile delivery device 110 that the user device 108 making the inventory purchase request would like to engage in a purchase and delivery transaction with the mobile delivery device 110. The mobile delivery device 110 would then take action to facilitate the delivery and sale of the item of inventory”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Langan et al. as modified by Rodriguez et al. to include determining to deliver the item based on the information relating to the item and confirmation of delivery of the item as taught by Esmailzadeh to allow remote users to order goods from any number of mobile delivery vehicles to make activities in the day to day lives of users more convenient, secure, and efficient.
But Langan et al. as modified by Rodriguez et al. and Esmailzadeh keep silent for teaching updating an inventory database upon confirmation of delivery of the item.
In related endeavor, Jones et al. teach updating an inventory database upon confirmation of delivery of the item (par 0003, “Inventory management systems of such retail sales facilities generally update the perpetual inventory in their inventory management database based on the time of delivery. Such updates are made on an assumption that the products are delivered within the scheduled time window and that the products are binned and worked to the sales floor immediately or shortly after the products are delivered”, par 0036, “FIG. 4 shows one embodiment of an operational workflow in implementing a method 400 in order to determine whether to update the electronic data in the inventory management database 140 representing the perpetual inventory of products 190 at the retail sales facility 110 and if so, a delay period determined based on the processed factors associated with the products 190 “, par 0039, “If the electronic data retrieved from the inventory management database 140 indicates that the picks of the cold chain product 190 off the delivery truck were complete, the control unit 210 is programmed to send a signal to the inventory management database to update the perpetual inventory (step 420) “, par 0044-0045, “If the electronic data retrieved from the inventory management database 140 indicates that a product 190 was delivered outside of the projected time for delivery, the control unit 210 is programmed to send a signal to the inventory management database 140 to delay the updating of the perpetual inventory or to update the perpetual inventory (step 420) as appropriate”)
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing data of the claimed invention to modified Langan et al. as modified by Rodriguez et al. and Esmailzadeh to include updating an inventory database upon confirmation of delivery of the item as taught by Jones et al. to update perpetual inventory at a retail sales facility after products are delivered to a retail sales facility to provide precisely and efficiently inventory management to reduce the costs associated with product sorting at the retail sales facility and increase the productivity of the workers at the retail sales facility.
Regarding claims 18-20 and 22-26, Langan et al. as modified by Rodriguez et al., Esmailzadeh and Jones et al. teach all the limitation of claim 17, and Langan et al. further teach claims 18-20 and 22-26 (the claims 18-20 and 22-26 are similar in scope to claims 2 or 3, or 4, 8-9, 7, and 11-14 and are rejected under the same rational).
Claim(s) 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2021/0192162 to Rodriguez et al. in view of U.S. PGPubs 2018/0181907 to Esmailzadeh, further in view of U.S. PGPubs 2017/0161674 to Jones et al., further in view of U.S. PGPubs 2023/0238098 to Langan et al., further in view of U.S. Patent 11941679 to Manyam.
Regarding claim 27, Rodriguez et al. as modified by Esmailzadeh, Jones et al., and Langan et al. all the limitation of claim 26, the claim 27 is similar in scope to claim 15 and is rejected under the same rational.
Claim(s) 28 is/are rejected under 35 U.S.C. 103 as being unpatentable over U.S. PGPubs 2021/0192162 to Rodriguez et al. in view of U.S. PGPubs 2018/0181907 to Esmailzadeh, further in view of U.S. PGPubs 2017/0161674 to Jones et al., further in view of U.S. PGPubs 2023/0238098 to Langan et al., further in view of U.S. Patent 11941679 to Manyam, further in view of U.S. PGPubs 2007/0021964 to Maenishi et al..
Regarding claim 28, Rodriguez et al. as modified by Esmailzadeh, Jones et al., Langan et al., and Manyam teach all the limitation of claim 27, the claim 28 is similar in scope to claim 16 and is rejected under the same rational.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jin Ge whose telephone number is (571)272-5556. The examiner can normally be reached 8:00 to 5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571)272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JIN . GE
Examiner
Art Unit 2619
/JIN GE/Primary Examiner, Art Unit 2619
Prosecution Insights