DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 – 17 are rejected under 35 U.S.C. 103 as being unpatentable over Fisher et al. (Publication: US 2023/0079388 A1) in view of Ogasawara et al. (Publication: US 2022/0277511 A1).
Regarding claim 1, Fisher discloses an information processing device comprising ([0058], [0133] – system 101, memory stores instruction and executed by the processor to perform the following methods: ):
circuitry configured to ([0058], [0133] – system 101, memory stores instruction and executed by the processor to perform the following methods: ):
control a position and an angle of a scanner and scan an appearance of a subject installed on an installation base to generate a scan image ([0091] - Cameras 114 are placed and oriented such that areas of the floor 220 and shelves can be seen by at least two cameras. The cameras 114 also cover floor space in front of the shelve unit A 402 and shelve unit B 404. Camera angles are selected to have both steep perspective, straight down, and angled perspectives that give more full body images of the customers (subjects). In one example embodiment, the cameras 114 are configured at an eight (8) foot height or higher throughout the shopping store.);
generate three-dimensional model data from the scan image ([0040] FIG. 7B presents a pair of images of a same area of real space captured by two sensors and identification of three pairs of points for determining an inventory item's position in three-dimensional area of real space.);
generate an image in which a three-dimensional model based on the three- dimensional model data is disposed in a [[virtual space]] ([0298] As the technology disclosed includes logic to create three-dimensional poses of subjects and display structures in the area of real space, the technology disclosed can present three-dimensional grids on inventory display structures as shown in FIG. 13A. The information regarding placement of items in the area of real space as included in planograms can be given as input to the two camera view tool. The technology disclosed includes logic to use that planogram information and the 3D grid of shelves to create a camogram as shown in FIG. 13B. In such an implementation, the technology disclosed uses 3D grids and planograms to draw boxes on the inventory display structures by using the geometry of the 3D grids and the information in the planograms as shown in FIG. 13C.); and
receive input of attribute information related to the subject to be scanned and performance of the subject ( [0130] - Fig. 5A - there is no bottle in the cell in 506 Fig. 5 A, “input”
[0307] that shelves have item slots (or cells or shelf) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant, “input”. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. Empty slot/shelf “performance” In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram.),
wherein the circuitry is configured to generate the image so as to include the performance for each subject according to the attribute information (
[0307] The process to map grids on camogram can start with generating an image of a shelf. The generated image may be of an empty shelf or an empty fridge. That shelves have item slots (or cells or shelf) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant, “input”. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. Empty slot/shelf “performance” ).
Fisher does not disclose; however Ogasawara discloses
virtual space ([0029] – generate a virtual space.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Fisher with virtual space as taught by Ogasawara. The motivation for doing is to have the space to provide more information about the scene.
Regarding claim 2, Fisher in view of Ogasawara disclose all the limitations of claim 1.
Fisher discloses wherein the attribute information includes first attribute information representing a type of the subject and second attribute information related to the performance, and the circuitry is configured to generate the image so as to include the performance for each subject according to the first attribute information and the second attribute information ([0130] - Fig. 5A - there is no bottle in the cell in 506 Fig. 5 A, “second attribute information includes a type of the second subject used for the performance”
[0307] that shelves have item slots (or cells or shelf) in which items are positioned and using that map the technology disclosed can identify the slots “a first subject” in which items are positioned and which slots are vacant. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. Empty slot/shelf “performance” In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “second attribute information” ).
Regarding claim 3, Fisher in view of Ogasawara disclose all the limitations of claim 2.
Fisher discloses wherein the second attribute information includes information indicating a type of the performance ([0307] that shelves have item slots (or cells or shelf) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant. Empty slot/shelf or not Empty slot/shelf “type of performance” In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “each type of performance includes a different performance image” [0130] - Fig. 5A - there is no bottle in the cell in 506 Fig. 5 A, “type of performance”. ).
Regarding claim 4, Fisher in view of Ogasawara disclose all the limitations of claim 3.
Fisher discloses wherein the type of performance includes one or more types of performance among a plurality of different types of performance, and each type of performance includes a different performance image ([0307] that shelves have item slots (or cells or shelf) in which items are positioned and using that map the technology disclosed can identify the slots “a first subject” in which items are positioned and which slots are vacant. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. Empty slot/shelf or not Empty slot/shelf “performance” In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “each type of performance includes a different performance image” ).
Regarding claim 5, Fisher in view of Ogasawara disclose all the limitations of claim 4.
Fisher discloses wherein the subject is a subject among a first subject and a second subject, and the second subject is an accessory of the first subject, the first attribute information represents a type of the first subject, and the second attribute information includes a type of the second subject used for the performance (
[0130] - Fig. 5A - there is no bottle in the cell in 506 Fig. 5 A, “second attribute information includes a type of the second subject used for the performance”
[0307] that shelves have item slots (or cells or shelf) in which items are positioned and using that map the technology disclosed can identify the slots “a first subject” in which items are positioned and which slots are vacant. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. Empty slot/shelf “performance” In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “second attribute information” ).
Regarding claim 6, Fisher in view of Ogasawara disclose all the limitations of claim 5.
Fisher discloses wherein the second attribute information indicates that the second subject is present by indicating the type of the accessory (
[0307] that shelves have item slots (or cells) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “indicating the type of the accessory”. ), and
the second attribute information indicates that the second subject is not present by indicating that a part of the first subject is used for the performance (
[0307] The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. Empty slot/shelf “performance”. In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge, items is not present, “second subject is not present”.
[0130], [0162], Fig. 6 - in 620 , when there is changed detected, meaning there is no bottle in the cell in 506 Fig. 5 A, “the second subject is not present”).
Regarding claim 7, Fisher in view of Ogasawara disclose all the limitations of claim 6.
Fisher discloses wherein the circuitry is configured to perform processing of associating the second subject with a predetermined position of the first subject from a first scan image of the first subject and a second scan image of the second subject ( [0017], [0138] - Fig. 5A - The received signal indicating the change is generated by matching at least one factored image of inventory locations in the area of real space captured in a first time interval with at least one factored image of inventory locations in the area of real space captured in a second time interval. The factored images are generated by removing foreground objects occluding inventory items placed in inventory locations. The bottle is visible or not visible in the shelf 506, “a predetermined position” ), and
generate the three-dimensional model data such that the first subject holds the second subject ( [0162], [0166], [0195] - Fig. 6 if no change is detected 620 on the shelf then go back to capture, generate, the image. Since no change is detected on the shelf meaning shelf has the bottle.
[0089] The cameras 114 are arranged to track subjects (or entities) in a three dimensional (abbreviated as 3D) real space. In the example embodiment of the shopping store, the real space can include the area of the shopping store where items for sale are stacked in shelves. A point in the real space can be represented by an (x, y, z) coordinate system. Each point in the area of real space for which the system is deployed is covered by the fields of view of two or more cameras 114.).
Regarding claim 8, Fisher in view of Ogasawara disclose all the limitations of claim 7.
Fisher discloses wherein the second attribute information further indicates the presence or absence of the second subject (
[0130] - Fig. 5A - there is no bottle in the cell in 506 Fig. 5 A, “the second subject is not present” ),
the scan image of the second subject is not generated when the second attribute information indicates that the second subject is not present ( [0130], [0162], Fig. 6 - in 620 , when there is changed detected, meaning there is no bottle in the cell in 506 Fig. 5 A, “the second subject is not present”, then it processed to step 534 without capturing the image again thus “the scan image of the second subject is not generated”
PNG
media_image1.png
720
544
media_image1.png
Greyscale
), and
the circuitry is configured to generate the three-dimensional model data from the first scan image without performing the processing of associating the second subject with the predetermined position of the first subject (
[0130], [0307] – The shell is empty and detection of the inventory event then triggers the process to update the map of the area of real space (e.g., the camogram) accordingly. The updated map does not include any item located in the cell 506.
[0074] FIG. 2 - camogram superimposed on the shelves or inventory display structures. The camogram can be considered as a map of inventory items placed in the area of real space, “without performing the processing of associating the second subject with the predetermined position of the first subject” because “no bottle in the predetermined position of the cell in 506 Fig. 5 A”. It is empty so there is no processing of performing associated with the bottle, second object.).
Regarding claim 9, Fisher in view of Ogasawara disclose all the limitations of claim 8.
Fisher discloses wherein the scan image of the second subject is generated when the second attribute information indicates that the second subject is present (
[0307] that shelves have item slots (or cells) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “so if the product is present, the subjects is generated”.).
Regarding claim 10, Fisher in view of Ogasawara disclose all the limitations of claim 9.
Fisher discloses to differentiate a first scanning range scanned when the second subject is installed on the installation base from a second scanning range scanned when the second subject is not installed on the installation base ([0131] changes in the area of real space, can cause an image captured of an inventory display structure to change at a current time interval as compared to the image of the same inventory display structure in a previous time interval.
[0158] - The model detects a “change” if there is an item with a new SKU placed in that area, or the item with the same SKU as in the previous image has changed positions “when the second subject is not installed on the installation base” or orientation at the same location, or there is a new occlusion, and the model can no longer be confident in the classification decision.
), and
the first scanning range is larger than the second scanning range ([0173] - The four images in FIG. 7D (i.e., images 730, 732, 734, and 736) illustrate four warped images that are taken from different positions of the virtual viewpoint. The virtual viewpoint is positioned at a greater height when image 730 is captured, therefore a portion of the area of real space above the inventory display structure is also included in this image. The virtual viewpoint is moved to a lower height and is moved leftward for capturing image 732, thus including a larger portion of the inventory display structure. The height of the virtual viewpoint is moved further lower and is moved rightward when capturing the image 734.).
Regarding claim 11, Fisher in view of Ogasawara disclose all the limitations of claim 10.
Fisher discloses configured to detect whether the second subject is installed on the installation base, and the circuitry does not receive the second attribute information indicating the type of the accessory, when installation of the second subject is not detected (
[0130], [0162], Fig. 6 - in 620 , when there is changed detected, meaning there is no bottle in the cell in 506 Fig. 5 A, “does not receive the second attribute information indicating the type of the accessory”, then it processed to step 534 without capturing the image again .
[0307] that shelves have item slots (or cells) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, “first subject”, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. In other words, in this method the process to map grids on camogram can start with an image of a shelf. ).
Regarding claim 12, Fisher in view of Ogasawara disclose all the limitations of claim 11.
Fisher discloses wherein the first subject is a model, and the second subject is an accessory part of the model (
[0130] - Fig. 5A - there is no bottle “second subject is an accessory part” in the cell in 506 Fig. 5 A.
[0307] that shelves have item slots (or cells) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, “first subject”, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “indicating the type of the accessory”.).
Ogasawara discloses
First subject is a doll-shaped model ([0019] – the article targeted for image capture is any of a plastic model that can be assembled, an action figure (figure with a movable joint portion), a toy body, a doll, and the like, which are collectively referred to as “models”.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Fisher in view Ogasawara with First subject is a doll-shaped model as taught by Ogasawara. The motivation for doing is to have the figure to provide more information about the scene.
Regarding claim 13, Fisher in view of Ogasawara disclose all the limitations of claim 12.
Fisher discloses includes information on a color used for the performance ([0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “indicating the type of the accessory”.).
Regarding claim 14, Fisher in view of Ogasawara disclose all the limitations of claim 13.
Fisher discloses configured to notify of an operation state of the information processing device, wherein the circuitry is configured to notify that the information processing device is in at least one of a state in which the information processing device is receiving the input of the attribute information and a state in which the scanning is being performed (
[0130] - Fig. 5A - there is no bottle in the cell in 506 Fig. 5 A, “input”
[0307] that shelves have item slots (or cells or shelf) in which items are positioned and using that map the technology disclosed can identify the slots in which items are positioned and which slots are vacant, “input”. If there is a shelf or a portion of the shelf that is empty then using the planned placement of items in the slots in the shelf, it can be determined which items have been taken by one or more subjects or which items have been placed in the shelf in a current time interval. The slots can be compared across time intervals to determine which items have been taken and which items have been placed in slots that were empty in the previous time interval. Empty slot/shelf “performance” In other words, in this method the process to map grids on camogram can start with an image of a shelf. The image may be of an empty shelf or an empty fridge. [0303] FIG. 14B illustrates search of item using camogram. The reviewer can select all “GatoradeTM” products to be shown in a particular colored boxes such as blue or yellow color etc, “second attribute information” ).
Regarding claim 15, Fisher in view of Ogasawara disclose all the limitations of claim 1.
Ogasawara discloses to transmit data of the generated image to a display device to enable the display device to display the generated image ([0038] - a virtual viewpoint image generated by the image generating apparatus 104 or image information related thereto as a second output product, and displays the virtual viewpoint image or the image information. Details of this processing will be described later using the drawings.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Fisher in view Ogasawara with to transmit data of the generated image to a display device to enable the display device to display the generated image as taught by Ogasawara. The motivation for doing is to have the transmission to provide more information about the scene.
Regarding claim 16, Fisher in view of Ogasawara disclose all the limitations of claim 2.
Fisher discloses configured to control the position and the angle of the scanner to change a range scanned by the scanner according to the first attribute information (
[0069] – cameras image the shelf/cell “first attribute”.
[0091] Cameras 114 are placed and oriented such that areas of the floor 220 and shelves can be seen by at least two cameras. The cameras 114 also cover floor space in front of the shelve unit A 402 and shelve unit B 404. Camera angles are selected to have both steep perspective, straight down, and angled perspectives that give more full body images of the customers (subjects). In one example embodiment, the cameras 114 are configured at an eight (8) foot height or higher throughout the shopping store. In one embodiment, the area of real space includes one or more designated unmonitored locations such as restrooms.
).
Regarding claim 17, see rejection on claim 1.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MING WU whose telephone number is (571)270-0724. The examiner can normally be reached on Monday - Thursday and alternate Fridays: 9:30am - 6:00pm EST .
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING WU/
Primary Examiner, Art Unit 2618
