DETAILED ACTION
Status of Claims
This action is in reply to the Applicant Remarks and Amendments filed on 03/02/2026.
Claims 1, 8, 10, and 14-19 have been amended.
Claims 1-20 are currently pending and have been examined.
This action is made FINAL.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments, see Page 17, filed 03/02/2026, with respect to the 35 U.S.C. 112(b) rejection of Claim 19 have been fully considered and are persuasive. The 35 U.S.C. 112(b) rejection of Claim 19 has been withdrawn.
Applicant’s arguments, see Pages 8-16, filed 03/02/2026, with respect to the 35 U.S.C. 101 rejection of Claims 1-20 have been fully considered and are persuasive. The 35 U.S.C. 101 rejection of Claims 1-20 has been withdrawn. Examiner notes that the amended independent claims 1, 8, and 16 overcome the 35 U.S.C. 101 rejection. Particularly, the additional elements in the amended independent claims and the specific order of combination of claim limitations integrate the judicial exception into a practical application because they apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception – (See MPEP 2106.05(e) and Vanda Memo).
Applicant’s arguments, see Pages 17-23, filed 03/02/2026, with respect to the 35 U.S.C. 102 rejection of Claims 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument..
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-2, 5, and 7 are rejected under 35 U.S.C. 102(a)(1) as being clearly anticipated by SHAKES et al. (US PG Pub. No. 2015/0221021 A1; hereinafter "SHAKES").
Regarding Claim 1, SHAKES teaches a packing station monitoring system comprising: a support arm configured to be mounted proximate to an active manufacturing packing station where cartons are being packed; a content capture device coupled to the support arm, the content capture device comprising a camera; a computing device in communication with the content capture device (See Figs. 2, 3A, and 3B, “FIG. 2 illustrates one embodiment of an exemplary physical layout of an order fulfillment center. Items for an order may be moved from inventory 30, through sorting stations 50, to one or more packing stations 60. The order fulfillment center illustrated in FIG. 2 includes various data capture devices, such as image capture devices, according to one embodiment. For example, an order fulfillment center may include one or more cameras or other image capture devices 310 configured to capture images of order processing at one or more processing stations, such as sorting stations 50, packing stations 60, and shipping stations 70, according to some embodiments. In one embodiment, all image capture devices 310 may represent still image capture devices, while in other embodiments, one or more of image capture devices 310 may represent video capture devices.” in Paragraph [0043], “FIG. 3A illustrates the capturing of images for an order being processed at an order fulfillment center, in one embodiment. For instance, a packing agent 320 may package one or more items 330 into shipping carton 350, in one embodiment. In one embodiment, …, while in other embodiments, the image capture devices may be suitably situated to capture images of the processing of an order. In some embodiments, the image capture devices 310 may be rigidly fixed in place, while in other embodiments, they may be configured to move or pan about in order to better capture images of the entire order processing.” in Paragraph [0053], and “FIG. 3B illustrates another embodiment of an order being processing at an order fulfillment center. While FIG. 3A, discussed above, illustrates single packing agent 320 and a single image capture device 310, FIG. 3B illustrates a station or stations where multiple processing agents 310 each perform one part of processing an order and multiple image capture devices 310 may be utilized in the capturing of images of the order processing. Thus, processing agent 320 may pack ordered items 330 into a shipping carton 350, processing agent 320b may insert packing material 335, such as foam peanuts or shredded paper, into shipping carton 350 around items 330, processing agent 320c may insert collateral materials 340, such as a receipt, packing slip, flyers, and/or advertisements, into shipping carton 350, and processing agent 320d may seal, address and weigh shipping carton 350, according to one embodiment. In other embodiments, different numbers of processing agents may perform different processing functions or perform them in a different order than illustrated by FIG. 3B.” in Paragraph [0060]), the computing device comprising: a processor; a memory storing instructions that, when executed by the processor, cause the computing device to (See “FIG. 13 is a block diagram illustrating an embodiment of a computer system usable to implement visual verification of order processing. In one embodiment, an order fulfillment control system, such as control system 300, illustrated in FIG. 3A, may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media, such as computer system 1300 illustrated in FIG. 13.” in Paragraph [0112], “In the illustrated embodiment, computer system 1300 may include one or more processors 1310 coupled to a system memory 1320 via an input/output (I/O) interface 1330.” in Paragraph [0113], and Fig. 13): receive an identifier associated with a carton to be packed at the active manufacturing packing station (See “As illustrated by block 400, one or more items for an order may arrive at a packing station.” in Paragraph [0065], “For instance, in one embodiment, every order may have an identification code on order paperwork that may be scanned, read, manually entered, or otherwise associated with the order during processing. Alternatively, a processing agent may read, scan, or manually enter an order ID code before starting to process an order and that order ID may be associated with any images captured during the processing of that order.” in Paragraph [0045], and “In some embodiments, processing agents may use identification codes when processing orders. For example, identification codes on individual items or on paperwork associated with an order may be read, either manually or via a reading device, as the order is being processed. In one embodiment, control system 300 may use the identification codes as part of tracking the progress of an order through a processing station or through the order fulfillment center. In other embodiments, the reading of a identification code may itself trigger the capturing of images. Additionally, in some embodiments, captured images may be associated with a identification code for an order. In some embodiments, processing agents may use devices configured to read or scan identification codes, either from individual items, order paperwork, or both. In other embodiments, however, processing agents may manually enter identification codes.” in Paragraph [0063]); decode the identifier to retrieve metadata associated with the carton (See “In another example, a radio frequency identification device (RFID) may be associated with an item during order processing and may be detected, either automatically or manually, as the item enters and/or exits various stages of order processing. For instance, an RFID may be temporarily attached to an item or more permanently attached to (or incorporated into or with) an item, according to different embodiments. Detecting an RFID may trigger the capturing of visual verification data characteristic of the particular stage of order processing. For example, an item may be detected, (e.g. by detecting an RFID, reading a scan code, or visually by processing personnel) upon arrival at a particular stage of order processing, and one or more types of data (e.g. images, audio, environmental, timing, etc) may be captured. Additionally, in other embodiments, information usable to associate a particular item with an order, or to associate a particular order with a customer, may be automatically collected. For instance, in one embodiment, a RFID may provide information allowing control system 300 to identify an item, match an item to a corresponding order, and/or match an order with a customer.” in Paragraph [0046] and “In other embodiments, however, the packing personal processing an order may utilize a scanning device to read an identification code associated with the order throughout the processing of the order and thus control system 300 may be configured to receive the identification code data and use it to match captured images with the correct order.” in Paragraph [0068]); initiate content capture by the content capture device for a time duration during packing operations at the active manufacturing packing station (See “FIG. 3B also illustrates multiple motion detecting devices 500, or other sensors, that may be configured to trigger data capture as an order progresses through the packing station. In some embodiments, each motion detection device 500 may be coupled to one or more individual data capture devices, such as image capture devices 310, such that data are captured based upon motion detected by the motion detection device. For example, motion detection device 500 may be coupled to image capture device 310 may detect the motion caused by a packing agent packing shipping carton 350 with ordered items 330 and may initiate capture of one or more images of the agent packing shipping carton 350. In other embodiments, however, control system 300 may receive indications from each motion detector 500 and may coordinate the capturing of data, such as images, based on the received signals from the motion detectors 500.” in Paragraph [0061], “Thus, as illustrated in FIG. 3B, multiple processing agents may process an order and images may be captured of the order being processed in any of a number of different manners, according to different embodiments. For instance, in one embodiment, agent 320a may pack individual items 330 into a shipping carton 350 and may manually trigger the capturing of images of each item being packed into the shipping carton. In another embodiment, motion detector 500a may trigger the capturing of images of items 330 being packed into shipping carton 350 by processing agent 320a. For example, the physical activity of packing the items 330 into shipping carton 350 may trigger motion detector 500a to initiate the capturing of images.” in Paragraph [0064], and “In other embodiments, image or video capturing may be initiated automatically by control system 300, or another computer system configured to do so. For example, the order fulfillment center may include one or more motion detection devices in and around the packing station configured to detect the arrival of items for processing and/or packaging. In one embodiment, a motion detector, such as motion detector 500, illustrated in FIG. 3B and described above, may detect the arrival of one or more items to the packing station and may signal control system 300. In response, control system 300 may communicate with one or more image capture devices, such as camera(s) 310, to start and stop image capture, in one embodiment.” in Paragraph [0066]); stop content capture by the content capture device in response to at least one of: the time duration elapsing or receipt of a user-initiated command from a packing operator to end recording (See “For instance, in one embodiment, control system 300 may be configured to analyze the differences between consecutively captured images to determine when an order has left the packing station. In yet other embodiments, packing personnel may manually signal the completion of order processing for an order via one or more manual switches, such as buttons, levers, foot pedals, scan code readers, etc. For example, after processing an order, a processing agent may use a scan code reader to read an identification code on the packed and sealed order and control system 300 may receive the identification code and information indicating the completion of order processing for that order. In some embodiments, … and control system 300 may … receive motion detection or manual signals, in order to determine which of the images to associate with a particular order. Thus, control system 300, or another computer system, may determine when to start and stop image capture for an order via a number of different mechanisms.” in Paragraph [0070]); and upload a packing record comprising the captured content, the metadata, and any documentation associated with one or more packed cartons to a data store (See “Control system 300 may receive the captured images and store them in a captured image database and associate them with the order being processed… In one embodiment, control system 300 may capture images of the finished, sealed, addressed order being loaded on delivery vehicle 360.” in Paragraph [0058], “After an order has been placed, the order may be processed at an order fulfillment center, as described above… Control system 300 may, according to one embodiment, receive the captured images, associate them with the order, and store them in image database 610 where web server 600 may make them available over the Internet 110. For example, in one embodiment, a separate web page may be automatically and dynamically generated for each order, while in another embodiment, such a web page may only be generated once the customer attempts to access the images, as will be discussed below.” in Paragraph [0079], and “FIG. 7 illustrates an image database of captured images such as may be used to store images on web server 600, according to some embodiments. In other embodiments, however, such a database may store other types of captured data, such as audio, date/time, environmental data, or other data, either instead of or in addition to captured image data. In some embodiments, captured visual verification data may be processed or enhanced before provided to a customer or other interested party. For instance, additional textual or graphic information may be added to captured images before the images are made available to a customer, either by the image capture device that captured the image or by control system 300, according to various embodiments. For example, in one embodiment, the date and/or time that an image was captured may be added to the captured image. Additionally, a customer name, or ID may be added to a captured image. Alternatively, information such as shipping personnel's names, packing station identifiers, the shipping weight of a package, environmental data, product identifiers, and/or order identifiers may also be added to the captured images. In general, any type of additional data or information that is relevant to and/or characteristic of an aspect of order processing may be added to or included with captured visual verification data. Such additional information and/or indicators may be included in, inserted into, overlaid over, or otherwise added to captured videos and still images in any of numerous ways, as is well understood in the art.” in Paragraph [0085]).
Regarding Claim 2, SHAKES teaches all the limitations of Claim 1 as described above. SHAKES also teaches wherein the content capture device further comprises a code reader configured to read the identifier (See “For example, after processing an order, a processing agent may use a scan code reader to read an identification code on the packed and sealed order …” in Paragraph [0070]).
Regarding Claim 5, SHAKES teaches all the limitations of Claim 1 as described above. SHAKES also teaches an audio device configured to provide audible alerts related to operation of the content capture device (See “The order fulfillment center may also include one or more motion detection devices or other sensors coupled to the control system and/or image capture devices to automate the capturing of images, according to one embodiment. For example, in one embodiment, the order fulfillment center may utilize sound-activated motion detection devices, ...” in Paragraph [0044]).
Regarding Claim 7, SHAKES teaches all the limitations of Claim 1 as described above. SHAKES also teaches wherein the data store is a cloud-based server configured to provide access to the captured content and metadata for verification of packing operations (See “In other embodiments, the order fulfillment center may fulfill orders for one or more affiliated merchants and captured images of order processing may be provided to the affiliated merchants via direct electronic notification or via a web services interface to a network accessible server. Such affiliated merchants may provide the captured images or other visual verification data regarding order processing to their customers. Alternatively, the order fulfillment center may send electronic notification messages including or referencing visual verification data to customers on behalf of affiliated merchants.” in Paragraph [0036] and “Additionally, in some embodiments, verification data, such as captured images, may be reviewed at virtually any location, not just at the order fulfillment center. For example, captured images may be available over the Internet, or may be transferred via email or other electronic data communication to wherever they are to be reviewed. In some embodiments, captured images may be made available electronically over a network such that the images are randomly accessible over the network. Companies may take advantage of the fact that captured verification data may be reviewed anywhere in the world to employ personnel in different locations, such as close to customers, or where labor costs are less. Thus, in one embodiment, a customer service representative remote from an order fulfillment center may review captured images, or other verification data, related to a customer complaint. For example, customer service representatives located in Great Britain may review customer complaints for customers who live in England, even if the order was shipped from the United States.” in Paragraph [0042]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 3-4, 6, 16-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over SHAKES in view of Eckman et al. (US PG Pub. No. 2021/0133666 A1; hereinafter "Eckman").
Regarding Claim 3, SHAKES teaches all the limitations of Claim 1 as described above. Although SHAKES teaches “a general-purpose computer” (See Paragraph [0112]), SHAKES does not explicitly teach “a tablet computer”. However, Eckman teaches wherein the computing device comprises a tablet computer (See “For example, if a font style on the pallet 114 cannot be read or recognized by a text recognition module, then the user may access the pallet 114's profile from the user's computing device (e.g., smartphone, laptop, tablet) and manually input the text that appears on the pallet 114.” in Paragraph [0064]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include wherein the computing device comprises a tablet computer, as taught by Eckman, in order to provide a more user-friendly device.
Regarding Claim 4, SHAKES teaches all the limitations of Claim 1 as described above. Although SHAKES teaches the areas of the facility where images may be captured may be arranged and lighted in ways to enhance the final captured images (See Paragraph [0050]), SHAKES does not explicitly teach “wherein the support arm comprises a light to illuminate an area below the support arm”. However, Eckman teaches wherein the support arm comprises a light to illuminate an area below the support arm (See “FIG. 2I depicts an example configuration of the scanning frame 200 including one or more light sources. In the present example, light sources 230A and 230B (e.g., light bars) are attached to the scanning frame 200, and extend parallel to the height of the frame along each side.” in Paragraph [0080] and Fig. 2I).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include wherein the support arm comprises a light to illuminate an area below the support arm, as taught by Eckman, in order to provide consistent lighting such that surfaces of packages are consistently illuminated, thus facilitating the capture of better quality images (See Paragraph [0080] of Eckman).
Regarding Claim 6, SHAKES teaches all the limitations of Claim 1 as described above. SHAKES does not explicitly teach; however, Eckman teaches wherein the computing device further comprises a lockdown application configured to limit access to applications other than an application for controlling the content capture device (See “Thus, for example, expansion memory 2474 may be provided as a security module for device 2450, and may be programmed with instructions that permit secure use of device 2450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.” in Paragraph [0227] and Fig. 24).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include wherein the computing device further comprises a lockdown application configured to limit access to applications other than an application for controlling the content capture device, as taught by Eckman, in order to make the system more efficient and effective.
Regarding Claim 16, SHAKES teaches a method for monitoring packing operations at an active manufacturing packing station, the method comprising: mounting a support arm above the active manufacturing packing station; removably coupling a … computing device to the support arm, the … computing device comprising a camera (See Figs. 2, 3A, and 3B, “FIG. 2 illustrates one embodiment of an exemplary physical layout of an order fulfillment center. Items for an order may be moved from inventory 30, through sorting stations 50, to one or more packing stations 60. The order fulfillment center illustrated in FIG. 2 includes various data capture devices, such as image capture devices, according to one embodiment. For example, an order fulfillment center may include one or more cameras or other image capture devices 310 configured to capture images of order processing at one or more processing stations, such as sorting stations 50, packing stations 60, and shipping stations 70, according to some embodiments. In one embodiment, all image capture devices 310 may represent still image capture devices, while in other embodiments, one or more of image capture devices 310 may represent video capture devices.” in Paragraph [0043], “FIG. 3A illustrates the capturing of images for an order being processed at an order fulfillment center, in one embodiment. For instance, a packing agent 320 may package one or more items 330 into shipping carton 350, in one embodiment. In one embodiment, …, while in other embodiments, the image capture devices may be suitably situated to capture images of the processing of an order. In some embodiments, the image capture devices 310 may be rigidly fixed in place, while in other embodiments, they may be configured to move or pan about in order to better capture images of the entire order processing.” in Paragraph [0053], and “FIG. 3B illustrates another embodiment of an order being processing at an order fulfillment center. While FIG. 3A, discussed above, illustrates single packing agent 320 and a single image capture device 310, FIG. 3B illustrates a station or stations where multiple processing agents 310 each perform one part of processing an order and multiple image capture devices 310 may be utilized in the capturing of images of the order processing. Thus, processing agent 320 may pack ordered items 330 into a shipping carton 350, processing agent 320b may insert packing material 335, such as foam peanuts or shredded paper, into shipping carton 350 around items 330, processing agent 320c may insert collateral materials 340, such as a receipt, packing slip, flyers, and/or advertisements, into shipping carton 350, and processing agent 320d may seal, address and weigh shipping carton 350, according to one embodiment. In other embodiments, different numbers of processing agents may perform different processing functions or perform them in a different order than illustrated by FIG. 3B.” in Paragraph [0060]); receiving, via the camera, an image of a … code identifying the specific manufacturing packing station affixed to a carton to be packed (See “As illustrated by block 400, one or more items for an order may arrive at a packing station.” in Paragraph [0065], “Video or images, as well as other data, may be recorded or captured using any of a number of different techniques, as is understood in the art. For example, in one embodiment, standard, off-the-shelf cameras may be placed in appropriate places within a materials processing facility to capture one or more aspects of order processing… For instance, in one embodiment, every order may have an identification code on order paperwork that may be scanned, read, manually entered, or otherwise associated with the order during processing. Alternatively, a processing agent may read, scan, or manually enter an order ID code before starting to process an order and that order ID may be associated with any images captured during the processing of that order.” in Paragraph [0045], and “In some embodiments, processing agents may use identification codes when processing orders. For example, identification codes on individual items or on paperwork associated with an order may be read, either manually or via a reading device, as the order is being processed. In one embodiment, control system 300 may use the identification codes as part of tracking the progress of an order through a processing station or through the order fulfillment center. In other embodiments, the reading of a identification code may itself trigger the capturing of images. Additionally, in some embodiments, captured images may be associated with a identification code for an order. In some embodiments, processing agents may use devices configured to read or scan identification codes, either from individual items, order paperwork, or both. In other embodiments, however, processing agents may manually enter identification codes.” in Paragraph [0063]); decoding the … code to retrieve metadata comprising a carton identifier, a sales order number, and a purchase order number associated with the carton (See “In another example, a radio frequency identification device (RFID) may be associated with an item during order processing and may be detected, either automatically or manually, as the item enters and/or exits various stages of order processing. For instance, an RFID may be temporarily attached to an item or more permanently attached to (or incorporated into or with) an item, according to different embodiments. Detecting an RFID may trigger the capturing of visual verification data characteristic of the particular stage of order processing. For example, an item may be detected, (e.g. by detecting an RFID, reading a scan code, or visually by processing personnel) upon arrival at a particular stage of order processing, and one or more types of data (e.g. images, audio, environmental, timing, etc) may be captured. Additionally, in other embodiments, information usable to associate a particular item with an order, or to associate a particular order with a customer, may be automatically collected. For instance, in one embodiment, a RFID may provide information allowing control system 300 to identify an item, match an item to a corresponding order, and/or match an order with a customer.” in Paragraph [0046] and “In other embodiments, however, the packing personal processing an order may utilize a scanning device to read an identification code associated with the order throughout the processing of the order and thus control system 300 may be configured to receive the identification code data and use it to match captured images with the correct order.” in Paragraph [0068]); initiating content capture by the camera for a predetermined time duration during active packing operations at the manufacturing packing station, wherein the content capture comprises capturing one or more of photo or video (See “FIG. 3B also illustrates multiple motion detecting devices 500, or other sensors, that may be configured to trigger data capture as an order progresses through the packing station. In some embodiments, each motion detection device 500 may be coupled to one or more individual data capture devices, such as image capture devices 310, such that data are captured based upon motion detected by the motion detection device. For example, motion detection device 500 may be coupled to image capture device 310 may detect the motion caused by a packing agent packing shipping carton 350 with ordered items 330 and may initiate capture of one or more images of the agent packing shipping carton 350. In other embodiments, however, control system 300 may receive indications from each motion detector 500 and may coordinate the capturing of data, such as images, based on the received signals from the motion detectors 500.” in Paragraph [0061], “Thus, as illustrated in FIG. 3B, multiple processing agents may process an order and images may be captured of the order being processed in any of a number of different manners, according to different embodiments. For instance, in one embodiment, agent 320a may pack individual items 330 into a shipping carton 350 and may manually trigger the capturing of images of each item being packed into the shipping carton. In another embodiment, motion detector 500a may trigger the capturing of images of items 330 being packed into shipping carton 350 by processing agent 320a. For example, the physical activity of packing the items 330 into shipping carton 350 may trigger motion detector 500a to initiate the capturing of images.” in Paragraph [0064], and “In other embodiments, image or video capturing may be initiated automatically by control system 300, or another computer system configured to do so. For example, the order fulfillment center may include one or more motion detection devices in and around the packing station configured to detect the arrival of items for processing and/or packaging. In one embodiment, a motion detector, such as motion detector 500, illustrated in FIG. 3B and described above, may detect the arrival of one or more items to the packing station and may signal control system 300. In response, control system 300 may communicate with one or more image capture devices, such as camera(s) 310, to start and stop image capture, in one embodiment.” in Paragraph [0066]); stopping content capture upon receipt of a user-initiated command from a packing operator or expiration of the predetermined time duration (See “For instance, in one embodiment, control system 300 may be configured to analyze the differences between consecutively captured images to determine when an order has left the packing station. In yet other embodiments, packing personnel may manually signal the completion of order processing for an order via one or more manual switches, such as buttons, levers, foot pedals, scan code readers, etc. For example, after processing an order, a processing agent may use a scan code reader to read an identification code on the packed and sealed order and control system 300 may receive the identification code and information indicating the completion of order processing for that order. In some embodiments, … and control system 300 may … receive motion detection or manual signals, in order to determine which of the images to associate with a particular order. Thus, control system 300, or another computer system, may determine when to start and stop image capture for an order via a number of different mechanisms.” in Paragraph [0070]); assembling a packing record comprising the captured content, the retrieved metadata, and any documentation associated with one or more packed cartons; initiating upload of the packing record to a cloud-based data store (See “Control system 300 may receive the captured images and store them in a captured image database and associate them with the order being processed… In one embodiment, control system 300 may capture images of the finished, sealed, addressed order being loaded on delivery vehicle 360.” in Paragraph [0058], “After an order has been placed, the order may be processed at an order fulfillment center, as described above… Control system 300 may, according to one embodiment, receive the captured images, associate them with the order, and store them in image database 610 where web server 600 may make them available over the Internet 110. For example, in one embodiment, a separate web page may be automatically and dynamically generated for each order, while in another embodiment, such a web page may only be generated once the customer attempts to access the images, as will be discussed below.” in Paragraph [0079], and “FIG. 7 illustrates an image database of captured images such as may be used to store images on web server 600, according to some embodiments. In other embodiments, however, such a database may store other types of captured data, such as audio, date/time, environmental data, or other data, either instead of or in addition to captured image data. In some embodiments, captured visual verification data may be processed or enhanced before provided to a customer or other interested party. For instance, additional textual or graphic information may be added to captured images before the images are made available to a customer, either by the image capture device that captured the image or by control system 300, according to various embodiments. For example, in one embodiment, the date and/or time that an image was captured may be added to the captured image. Additionally, a customer name, or ID may be added to a captured image. Alternatively, information such as shipping personnel's names, packing station identifiers, the shipping weight of a package, environmental data, product identifiers, and/or order identifiers may also be added to the captured images. In general, any type of additional data or information that is relevant to and/or characteristic of an aspect of order processing may be added to or included with captured visual verification data. Such additional information and/or indicators may be included in, inserted into, overlaid over, or otherwise added to captured videos and still images in any of numerous ways, as is well understood in the art.” in Paragraph [0085]); and managing sequential upload of multiple packing records to the cloud-based data store using a queuing process during concurrent packing operations (See “In one embodiment, control system 300 may initiate image capturing on a camera by camera basis, while in another embodiment, control system 300 may instruct all the cameras in the packing area to capture images concurrently. In another embodiment, packing station 60 may be equipped with both still image and video capture devices and control system 300 may initiate capturing of both still images and video for an order. In some embodiments, control system 300 may monitor, via multiple motion detectors 500, the movement of the items and/or package for an order through packing station 60 and may turn on and off individual image and/or video capture devices 310 as the items and package progress through the packing station.” in Paragraph [0067], “In other embodiments, however, all cameras in packing station 60 may be continually capturing images (or video) and control system 300 may utilize signals from motion detector(s) 500 to monitor an order being processed and match up (or associate) the captured images with the correct order. For instance, in some embodiments, a processing station may be continually processing orders one after another such that as one order is starting to be processed in one area, another order may be being packaged for shipment in another area. In such an embodiment, control system 300 may be able to determine how to correctly associate captured images with the appropriate order by utilizing the signals from one or more motion detector(s) 500. In other embodiments, however, the packing personal processing an order may utilize a scanning device to read an identification code associated with the order throughout the processing of the order and thus control system 300 may be configured to receive the identification code data and use it to match captured images with the correct order.” in Paragraph [0068], “After an order has been placed, the order may be processed at an order fulfillment center, as described above. One or more images, either still or video, may be captured of the order being processed. Control system 300 may, according to one embodiment, receive the captured images, associate them with the order, and store them in image database 610 where web server 600 may make them available over the Internet 110. For example, in one embodiment, a separate web page may be automatically and dynamically generated for each order, while in another embodiment, such a web page may only be generated once the customer attempts to access the images, as will be discussed below.” in Paragraph [0079], and Figs. 8A-8C).
As described above, SHAKES does not explicitly teach “a tablet” and “a QR code”, however, Eckman teaches a tablet (See “For example, if a font style on the pallet 114 cannot be read or recognized by a text recognition module, then the user may access the pallet 114's profile from the user's computing device (e.g., smartphone, laptop, tablet) and manually input the text that appears on the pallet 114.” in Paragraph [0064]) and a QR code (See “In some implementations, server 110 can include modules for each of the following parameter identifications: barcodes, QR codes, …” in Paragraph [0061]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include a tablet and a QR code, as taught by Eckman, in order to provide a more user-friendly device and to make the system more efficient and effective, respectively.
Regarding Claim 17, SHAKES in view of Eckman teaches all the limitations of Claim 16 as described above. SHAKES also teaches wherein the … code identifies the specific manufacturing packing station and is used to associate captured content with the particular manufacturing packing station where the packing operations occurred (See “In some embodiments, processing agents may use one or more manual switches to initiate the capturing of images. For example, in one embodiment, processing agents may manually initiate image, or video, capture when the items of an order arrive at the packing station. In other embodiments, however, processing agents may manually instigate the capturing of each image individually as the items are being processed and packaged for shipping. Any of a number of different types of suitable manual switches, such as buttons, toggle switches, levers, foot pedals, etc. may be used, in various embodiments, to initiate the capturing of images. In some embodiments, an identification code reader, such as a scan-code reader, may also serve as a manual image capture trigger. In other embodiments, image or video capturing may be initiated automatically by control system 300, or another computer system configured to do so. For example, the order fulfillment center may include one or more motion detection devices in and around the packing station configured to detect the arrival of items for processing and/or packaging. In one embodiment, a motion detector, such as motion detector 500, illustrated in FIG. 3B and described above, may detect the arrival of one or more items to the packing station and may signal control system 300. In response, control system 300 may communicate with one or more image capture devices, such as camera(s) 310, to start and stop image capture, in one embodiment.” in Paragraph [0066]).
SHAKES does not explicitly teach; however, Eckman teaches the QR code and the method further comprising: restricting access on the tablet computing device to applications other than a packing monitoring application using a lockdown application (See “Thus, for example, expansion memory 2474 may be provided as a security module for device 2450, and may be programmed with instructions that permit secure use of device 2450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.” in Paragraph [0227] and Fig. 24).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include wherein the computing device further comprises a lockdown application configured to limit access to applications other than an application for controlling the content capture device, as taught by Eckman, in order to make the system more efficient and effective.
Regarding Claim 18, SHAKES in view of Eckman teaches all the limitations of Claim 16 as described above. SHAKES also teaches adjusting a mount on the support arm to position the … computing device to capture content of an entire packing area (See “In some embodiments, the image capture devices 310 may be rigidly fixed in place, while in other embodiments, they may be configured to move or pan about in order to better capture images of the entire order processing.” in Paragraph [0053]).
As described above, SHAKES does not explicitly teach “a tablet”; however, Eckman teaches a tablet (See “For example, if a font style on the pallet 114 cannot be read or recognized by a text recognition module, then the user may access the pallet 114's profile from the user's computing device (e.g., smartphone, laptop, tablet) and manually input the text that appears on the pallet 114.” in Paragraph [0064]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include a tablet, as taught by Eckman, in order to provide a more user-friendly device.
Regarding Claim 20, SHAKES in view of Eckman teaches all the limitations of Claim 16 as described above. SHAKES also teaches associating each uploaded packing record with corresponding shipping information for the carton in the cloud-based data store (See “The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components.” in Paragraph [0235]); providing secure access to packing records for authorized users based on user authentication; and generating reports summarizing packing operations across multiple cartons (See “Control system 300 may send the customer an electronic notification, such as an email message, instant message, cell phone picture, text message, or other electronic notification, including one or more captured images of the order being processed, and/or the electronic notification may include a link, URL, or other reference usable to access the captured images. Control system 300 and/or web server 600 may apply various security and/or authentication measures to prevent unauthorized access to captured images of orders being processed, esp. in embodiments where a customer's name and/or address may be visible in one or more of the images. The electronic notification may also include information regarding any such security measures and may also, in some embodiment, include all or partial authentication credentials necessary for accessing captured images. For example, in one embodiment, the electronic notification may inform the customer to login to web server 600 using a name and password the customer had previously setup, such as when originally placing the order. In another embodiment, however, login information, or other authentication credential information may be included in the electronic notification, or in separate electronic notification.” in Paragraph [0080] and “In some embodiments, captured visual verification data may be processed or enhanced before provided to a customer or other interested party. For instance, additional textual or graphic information may be added to captured images before the images are made available to a customer, either by the image capture device that captured the image or by control system 300, according to various embodiments. For example, in one embodiment, the date and/or time that an image was captured may be added to the captured image. Additionally, a customer name, or ID may be added to a captured image. Alternatively, information such as shipping personnel's names, packing station identifiers, the shipping weight of a package, environmental data, product identifiers, and/or order identifiers may also be added to the captured images. In general, any type of additional data or information that is relevant to and/or characteristic of an aspect of order processing may be added to or included with captured visual verification data. Such additional information and/or indicators may be included in, inserted into, overlaid over, or otherwise added to captured videos and still images in any of numerous ways, as is well understood in the art. The modification or enhancement of captured images may be performed manually, automatically, or as a combination of both. As various methods of image manipulation are well known in the art, and as any suitable image manipulation method may be used to include additional information in captured images and/or video, those methods will not be discussed herein. In some embodiments, however, captured or collected data may not be enhanced so that a customer or other party is provided with raw, unaltered data.” in Paragraph [0085]).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over SHAKES in view of Eckman and Birkhofer et al. (US PG Pub. No. 2024/0381881 A1; hereinafter "Birkhofer").
Regarding Claim 19, SHAKES in view of Eckman teaches all the limitations of Claim 16 as described above. SHAKES in view of Eckman does not explicitly teach; however, Birkhofer teaches analyzing the captured content in real-time using an artificial intelligence model specially trained on manufacturing packing data to detect packing anomalies comprising at least one of: missing items, incorrect items, improper packing technique, or inadequate protective packaging during the packing process at the manufacturing packing station (See “Thus, upon starting at operation 585, the controller 230 receives image data from a vacuum sealed package at operation 590. The image data may be collected by computer vision systems 215A-215C. In some embodiments, the image data may include images, videos, and/or x-ray images. In other embodiments, the image data may include other types of image data discussed above. The description below is with respect to the data gathered by the computer vision system 215A on the conveyor table 205A. However, the description below is similarly applicable to the computer vision systems 215B-215C on the conveyor tables 205B-205C respectively. At operation 595, the controller 230 determines if there air or void inside the vacuum sealed package. In some embodiments, the controller 230, and particularly the data pre-processing engine 280 of the controller, may pre-process the image data. For example, in some embodiments, if multiple images (and/or videos) are captured for each piece of meat, the controller 230 may select an appropriate image (and/or video) for analysis (e.g., an image and/or video that shows a certain angle of the meat, shows a certain surface of the meat, etc.), combine multiple images (and/or videos) to obtain a suitable image and/or video (e.g., if none of the images are entirely suitable, the controller may stitch together multiple images to obtain a suitable image), modify images and/or videos (e.g., enhance the images to facilitate better analysis), convert video into static images, etc. In some embodiments, no pre-processing of the images (and/or videos) may be needed. Upon pre-processing (or if no pre-processing is used), the selected image data may be analyzed by the data analysis engine 285 of the controller 230. In some embodiments, the data analysis engine 285 may be previously trained to identify voids in the vacuum sealed package.” in Paragraph [0092], “In some embodiments, the computer vision system 215A may include a mass spectrometer that provides a mass spectrometer image. From the mass spectrometer image, the controller 230 may identify any residual gas or oxygen in the vacuum sealed package. For example, in some embodiments, the controller 230 may determine an outer boundary of the packaging material of the vacuum sealed package and an outer boundary of the meat inside the packaging material. If the outer boundary of the packaging material and the outer boundary of the meat inside the packaging material do not match or have a gap that is greater that a pre-determined threshold, the controller 230 may determine that a void exists in the vacuum sealed package. In other embodiments, the image data may include other types of images that allow the controller 230 to determine the presence of air or gap in the vacuum sealed package.” in Paragraph [0093], “In some embodiments, products are monitored exiting packaging machines and where the product type can be identified/confirmed and the integrity of the package can be evaluated (looking for air in the bag). Products that do not pass this inspection may be automatically delivered to a rework station to be unpackaged and returned to packaging... Once product makes it to case packing (packaged product is placed in a box), each box is monitored for the correct products and the correct number/weight of products. If the bar code scanning of the label on the box doesn't match the products that are inside the case, then box is automatically rejected to a rework station to be inspected and corrected. Rework may be monitored by employee and tracked by cause (too much product, wrong product, foreign objects, etc.) to create notifications to management depending on severity and frequency of occurrence.” in Paragraph [0115], and “When implemented as a machine learning or artificial intelligence engine, the data analysis engine 285 may be trained using training data to, for example, to classify data or identify underlying relationships in the data.” in Paragraph [0050]); and generating an alert displayed on a screen of the tablet computing device if a packing anomaly is detected during the active packing operations (See “Thus, upon training the data analysis engine 285 for detecting specific types of foreign objects, the image data from the operation 380 (or the image data that has been pre-processed) may be input into the data analysis engine. The data analysis engine may identify any foreign objects in that image data. At operation 390, the controller 230 determines if any foreign object is found in the meat. If yes, then at operation 395, the controller 230 stops the conveyor table 150A and raises an alert at operation 400.” in Paragraph [0072], “In some embodiments, an alert may be displayed on a dashboard associated with the conveyor table 150A (e.g., a dashboard positioned at the end of the conveyor table).” in Paragraph [0073], and “The smart manufacturing system 225 or at least portions thereof may be implemented in a variety of computing devices such as computers (e.g., …, tablets, …, or any other computing unit suitable for performing operations described herein.” in Paragraph [0040]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES in view of Eckman to include analyzing the captured content in real-time using an artificial intelligence model specially trained on manufacturing packing data to detect packing anomalies comprising at least one of: missing items, incorrect items, improper packing technique, or inadequate protective packaging during the packing process at the manufacturing packing station and generating an alert displayed on a screen of the tablet computing device if a packing anomaly is detected during the active packing operations, as taught by Birkhofer, in order to improve quality and customer’s satisfaction.
Claims 8-11 and 13-15 are rejected under 35 U.S.C. 103 as being unpatentable over SHAKES in view of Birkhofer.
Regarding Claim 8, SHAKES teaches a packing station monitoring system comprising: a support arm configured to be mounted proximate to a packing station where cartons are being packed; a content capture device coupled to the support arm, the content capture device comprising a camera and a code reader (See Figs. 2, 3A, and 3B, “FIG. 2 illustrates one embodiment of an exemplary physical layout of an order fulfillment center. Items for an order may be moved from inventory 30, through sorting stations 50, to one or more packing stations 60. The order fulfillment center illustrated in FIG. 2 includes various data capture devices, such as image capture devices, according to one embodiment. For example, an order fulfillment center may include one or more cameras or other image capture devices 310 configured to capture images of order processing at one or more processing stations, such as sorting stations 50, packing stations 60, and shipping stations 70, according to some embodiments. In one embodiment, all image capture devices 310 may represent still image capture devices, while in other embodiments, one or more of image capture devices 310 may represent video capture devices.” in Paragraph [0043], “FIG. 3A illustrates the capturing of images for an order being processed at an order fulfillment center, in one embodiment. For instance, a packing agent 320 may package one or more items 330 into shipping carton 350, in one embodiment. In one embodiment, …, while in other embodiments, the image capture devices may be suitably situated to capture images of the processing of an order. In some embodiments, the image capture devices 310 may be rigidly fixed in place, while in other embodiments, they may be configured to move or pan about in order to better capture images of the entire order processing.” in Paragraph [0053], “FIG. 3B illustrates another embodiment of an order being processing at an order fulfillment center. While FIG. 3A, discussed above, illustrates single packing agent 320 and a single image capture device 310, FIG. 3B illustrates a station or stations where multiple processing agents 310 each perform one part of processing an order and multiple image capture devices 310 may be utilized in the capturing of images of the order processing. Thus, processing agent 320 may pack ordered items 330 into a shipping carton 350, processing agent 320b may insert packing material 335, such as foam peanuts or shredded paper, into shipping carton 350 around items 330, processing agent 320c may insert collateral materials 340, such as a receipt, packing slip, flyers, and/or advertisements, into shipping carton 350, and processing agent 320d may seal, address and weigh shipping carton 350, according to one embodiment. In other embodiments, different numbers of processing agents may perform different processing functions or perform them in a different order than illustrated by FIG. 3B.” in Paragraph [0060], and “In some embodiments, an identification code reader, such as a scan-code reader, may also serve as a manual image capture trigger.” in Paragraph [0066]); a computing device in communication with the content capture device, the computing device comprising: a processor; a memory storing instructions that, when executed by the processor, cause the computing device to (See “FIG. 13 is a block diagram illustrating an embodiment of a computer system usable to implement visual verification of order processing. In one embodiment, an order fulfillment control system, such as control system 300, illustrated in FIG. 3A, may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media, such as computer system 1300 illustrated in FIG. 13.” in Paragraph [0112], “In the illustrated embodiment, computer system 1300 may include one or more processors 1310 coupled to a system memory 1320 via an input/output (I/O) interface 1330.” in Paragraph [0113], and Fig. 13): receive, via the code reader, an identifier associated with a carton to be packed at the active manufacturing packing station (See “As illustrated by block 400, one or more items for an order may arrive at a packing station.” in Paragraph [0065], “For instance, in one embodiment, every order may have an identification code on order paperwork that may be scanned, read, manually entered, or otherwise associated with the order during processing. Alternatively, a processing agent may read, scan, or manually enter an order ID code before starting to process an order and that order ID may be associated with any images captured during the processing of that order.” in Paragraph [0045], and “In some embodiments, processing agents may use identification codes when processing orders. For example, identification codes on individual items or on paperwork associated with an order may be read, either manually or via a reading device, as the order is being processed. In one embodiment, control system 300 may use the identification codes as part of tracking the progress of an order through a processing station or through the order fulfillment center. In other embodiments, the reading of a identification code may itself trigger the capturing of images. Additionally, in some embodiments, captured images may be associated with a identification code for an order. In some embodiments, processing agents may use devices configured to read or scan identification codes, either from individual items, order paperwork, or both. In other embodiments, however, processing agents may manually enter identification codes.” in Paragraph [0063]); decode the identifier to retrieve metadata associated with the carton (See “In another example, a radio frequency identification device (RFID) may be associated with an item during order processing and may be detected, either automatically or manually, as the item enters and/or exits various stages of order processing. For instance, an RFID may be temporarily attached to an item or more permanently attached to (or incorporated into or with) an item, according to different embodiments. Detecting an RFID may trigger the capturing of visual verification data characteristic of the particular stage of order processing. For example, an item may be detected, (e.g. by detecting an RFID, reading a scan code, or visually by processing personnel) upon arrival at a particular stage of order processing, and one or more types of data (e.g. images, audio, environmental, timing, etc) may be captured. Additionally, in other embodiments, information usable to associate a particular item with an order, or to associate a particular order with a customer, may be automatically collected. For instance, in one embodiment, a RFID may provide information allowing control system 300 to identify an item, match an item to a corresponding order, and/or match an order with a customer.” in Paragraph [0046] and “In other embodiments, however, the packing personal processing an order may utilize a scanning device to read an identification code associated with the order throughout the processing of the order and thus control system 300 may be configured to receive the identification code data and use it to match captured images with the correct order.” in Paragraph [0068]); initiate content capture by the camera, wherein the content capture comprises one or more of a photo capture or a video capture for a predetermined time duration during packing operations at the active manufacturing packing station (See “FIG. 3B also illustrates multiple motion detecting devices 500, or other sensors, that may be configured to trigger data capture as an order progresses through the packing station. In some embodiments, each motion detection device 500 may be coupled to one or more individual data capture devices, such as image capture devices 310, such that data are captured based upon motion detected by the motion detection device. For example, motion detection device 500 may be coupled to image capture device 310 may detect the motion caused by a packing agent packing shipping carton 350 with ordered items 330 and may initiate capture of one or more images of the agent packing shipping carton 350. In other embodiments, however, control system 300 may receive indications from each motion detector 500 and may coordinate the capturing of data, such as images, based on the received signals from the motion detectors 500.” in Paragraph [0061], “Thus, as illustrated in FIG. 3B, multiple processing agents may process an order and images may be captured of the order being processed in any of a number of different manners, according to different embodiments. For instance, in one embodiment, agent 320a may pack individual items 330 into a shipping carton 350 and may manually trigger the capturing of images of each item being packed into the shipping carton. In another embodiment, motion detector 500a may trigger the capturing of images of items 330 being packed into shipping carton 350 by processing agent 320a. For example, the physical activity of packing the items 330 into shipping carton 350 may trigger motion detector 500a to initiate the capturing of images.” in Paragraph [0064], and “In other embodiments, image or video capturing may be initiated automatically by control system 300, or another computer system configured to do so. For example, the order fulfillment center may include one or more motion detection devices in and around the packing station configured to detect the arrival of items for processing and/or packaging. In one embodiment, a motion detector, such as motion detector 500, illustrated in FIG. 3B and described above, may detect the arrival of one or more items to the packing station and may signal control system 300. In response, control system 300 may communicate with one or more image capture devices, such as camera(s) 310, to start and stop image capture, in one embodiment.” in Paragraph [0066]); stop content capture in response to at least one of: the time duration elapsing or receipt of a user-initiated stop command from a packing operator (See “For instance, in one embodiment, control system 300 may be configured to analyze the differences between consecutively captured images to determine when an order has left the packing station. In yet other embodiments, packing personnel may manually signal the completion of order processing for an order via one or more manual switches, such as buttons, levers, foot pedals, scan code readers, etc. For example, after processing an order, a processing agent may use a scan code reader to read an identification code on the packed and sealed order and control system 300 may receive the identification code and information indicating the completion of order processing for that order. In some embodiments, … and control system 300 may … receive motion detection or manual signals, in order to determine which of the images to associate with a particular order. Thus, control system 300, or another computer system, may determine when to start and stop image capture for an order via a number of different mechanisms.” in Paragraph [0070]); and upload a packing record comprising the captured content, the metadata, and any documentation associated with one or more packed cartons to a remote data store (See “Control system 300 may receive the captured images and store them in a captured image database and associate them with the order being processed… In one embodiment, control system 300 may capture images of the finished, sealed, addressed order being loaded on delivery vehicle 360.” in Paragraph [0058], “After an order has been placed, the order may be processed at an order fulfillment center, as described above… Control system 300 may, according to one embodiment, receive the captured images, associate them with the order, and store them in image database 610 where web server 600 may make them available over the Internet 110. For example, in one embodiment, a separate web page may be automatically and dynamically generated for each order, while in another embodiment, such a web page may only be generated once the customer attempts to access the images, as will be discussed below.” in Paragraph [0079], and “FIG. 7 illustrates an image database of captured images such as may be used to store images on web server 600, according to some embodiments. In other embodiments, however, such a database may store other types of captured data, such as audio, date/time, environmental data, or other data, either instead of or in addition to captured image data. In some embodiments, captured visual verification data may be processed or enhanced before provided to a customer or other interested party. For instance, additional textual or graphic information may be added to captured images before the images are made available to a customer, either by the image capture device that captured the image or by control system 300, according to various embodiments. For example, in one embodiment, the date and/or time that an image was captured may be added to the captured image. Additionally, a customer name, or ID may be added to a captured image. Alternatively, information such as shipping personnel's names, packing station identifiers, the shipping weight of a package, environmental data, product identifiers, and/or order identifiers may also be added to the captured images. In general, any type of additional data or information that is relevant to and/or characteristic of an aspect of order processing may be added to or included with captured visual verification data. Such additional information and/or indicators may be included in, inserted into, overlaid over, or otherwise added to captured videos and still images in any of numerous ways, as is well understood in the art.” in Paragraph [0085]).
SHAKES does not explicitly teach; however, Birkhofer teaches analyze the content capture in real-time using an artificial intelligence model specifically trained on manufacturing packing data to detect packing anomalies during the packing process (See “Thus, upon starting at operation 585, the controller 230 receives image data from a vacuum sealed package at operation 590. The image data may be collected by computer vision systems 215A-215C. In some embodiments, the image data may include images, videos, and/or x-ray images. In other embodiments, the image data may include other types of image data discussed above. The description below is with respect to the data gathered by the computer vision system 215A on the conveyor table 205A. However, the description below is similarly applicable to the computer vision systems 215B-215C on the conveyor tables 205B-205C respectively. At operation 595, the controller 230 determines if there air or void inside the vacuum sealed package. In some embodiments, the controller 230, and particularly the data pre-processing engine 280 of the controller, may pre-process the image data. For example, in some embodiments, if multiple images (and/or videos) are captured for each piece of meat, the controller 230 may select an appropriate image (and/or video) for analysis (e.g., an image and/or video that shows a certain angle of the meat, shows a certain surface of the meat, etc.), combine multiple images (and/or videos) to obtain a suitable image and/or video (e.g., if none of the images are entirely suitable, the controller may stitch together multiple images to obtain a suitable image), modify images and/or videos (e.g., enhance the images to facilitate better analysis), convert video into static images, etc. In some embodiments, no pre-processing of the images (and/or videos) may be needed. Upon pre-processing (or if no pre-processing is used), the selected image data may be analyzed by the data analysis engine 285 of the controller 230. In some embodiments, the data analysis engine 285 may be previously trained to identify voids in the vacuum sealed package.” in Paragraph [0092], “In some embodiments, products are monitored exiting packaging machines and where the product type can be identified/confirmed and the integrity of the package can be evaluated (looking for air in the bag). Products that do not pass this inspection may be automatically delivered to a rework station to be unpackaged and returned to packaging... Once product makes it to case packing (packaged product is placed in a box), each box is monitored for the correct products and the correct number/weight of products. If the bar code scanning of the label on the box doesn't match the products that are inside the case, then box is automatically rejected to a rework station to be inspected and corrected. Rework may be monitored by employee and tracked by cause (too much product, wrong product, foreign objects, etc.) to create notifications to management depending on severity and frequency of occurrence.” in Paragraph [0115], and “When implemented as a machine learning or artificial intelligence engine, the data analysis engine 285 may be trained using training data to, for example, to classify data or identify underlying relationships in the data.” in Paragraph [0050]); and generate an alert if a packing anomaly is detected (See “Thus, upon training the data analysis engine 285 for detecting specific types of foreign objects, the image data from the operation 380 (or the image data that has been pre-processed) may be input into the data analysis engine. The data analysis engine may identify any foreign objects in that image data. At operation 390, the controller 230 determines if any foreign object is found in the meat. If yes, then at operation 395, the controller 230 stops the conveyor table 150A and raises an alert at operation 400.” in Paragraph [0072] and “In some embodiments, an alert may be displayed on a dashboard associated with the conveyor table 150A (e.g., a dashboard positioned at the end of the conveyor table).” in Paragraph [0073]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include analyzing the content capture in real-time using an artificial intelligence model specifically trained on manufacturing packing data to detect packing anomalies during the packing process and generating an alert if a packing anomaly is detected, as taught by Birkhofer, in order to improve quality and customer’s satisfaction.
Regarding Claim 9, SHAKES in view of Birkhofer teaches all the limitations of Claim 8 as described above. SHAKES also teaches wherein the computing device and the content capture device are integrated into a … computer device (See “Image capture devices 310 may be coupled to control system 300 via network 100. Control system 300 may, in some embodiments, be configured to control and manage the image capture process for all orders being processed in the order fulfillment center.” in Paragraph [0043]). SHAKES does not explicitly teach “a single tablet computer device”; however, Birkhofer teaches a single tablet computer device (See “… in some embodiments, one or more of those engines may be integrated together into a single component and the single component may perform the operations of the individual components.” in Paragraph [0048] and “The smart manufacturing system 225 or at least portions thereof may be implemented in a variety of computing devices such as …, tablets, …, or any other computing unit suitable for performing operations described herein.” in Paragraph [0040]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include a single tablet computer device, as taught by Birkhofer, in order to make the system more efficient and effective.
Regarding Claim 10, SHAKES in view of Birkhofer teaches all the limitations of Claim 8 as described above. SHAKES does not explicitly teach; however, Birkhofer teaches wherein the artificial intelligence model is specifically trained to detect packing anomalies using manufacturing packing process data and is configured to perform real-time analysis during active packing operations (See “Thus, upon starting at operation 585, the controller 230 receives image data from a vacuum sealed package at operation 590. The image data may be collected by computer vision systems 215A-215C. In some embodiments, the image data may include images, videos, and/or x-ray images. In other embodiments, the image data may include other types of image data discussed above. The description below is with respect to the data gathered by the computer vision system 215A on the conveyor table 205A. However, the description below is similarly applicable to the computer vision systems 215B-215C on the conveyor tables 205B-205C respectively. At operation 595, the controller 230 determines if there air or void inside the vacuum sealed package. In some embodiments, the controller 230, and particularly the data pre-processing engine 280 of the controller, may pre-process the image data. For example, in some embodiments, if multiple images (and/or videos) are captured for each piece of meat, the controller 230 may select an appropriate image (and/or video) for analysis (e.g., an image and/or video that shows a certain angle of the meat, shows a certain surface of the meat, etc.), combine multiple images (and/or videos) to obtain a suitable image and/or video (e.g., if none of the images are entirely suitable, the controller may stitch together multiple images to obtain a suitable image), modify images and/or videos (e.g., enhance the images to facilitate better analysis), convert video into static images, etc. In some embodiments, no pre-processing of the images (and/or videos) may be needed. Upon pre-processing (or if no pre-processing is used), the selected image data may be analyzed by the data analysis engine 285 of the controller 230. In some embodiments, the data analysis engine 285 may be previously trained to identify voids in the vacuum sealed package.” in Paragraph [0092], “In some embodiments, the computer vision system 215A may include a mass spectrometer that provides a mass spectrometer image. From the mass spectrometer image, the controller 230 may identify any residual gas or oxygen in the vacuum sealed package. For example, in some embodiments, the controller 230 may determine an outer boundary of the packaging material of the vacuum sealed package and an outer boundary of the meat inside the packaging material. If the outer boundary of the packaging material and the outer boundary of the meat inside the packaging material do not match or have a gap that is greater that a pre-determined threshold, the controller 230 may determine that a void exists in the vacuum sealed package. In other embodiments, the image data may include other types of images that allow the controller 230 to determine the presence of air or gap in the vacuum sealed package.” in Paragraph [0093], “In some embodiments, products are monitored exiting packaging machines and where the product type can be identified/confirmed and the integrity of the package can be evaluated (looking for air in the bag). Products that do not pass this inspection may be automatically delivered to a rework station to be unpackaged and returned to packaging... Once product makes it to case packing (packaged product is placed in a box), each box is monitored for the correct products and the correct number/weight of products. If the bar code scanning of the label on the box doesn't match the products that are inside the case, then box is automatically rejected to a rework station to be inspected and corrected. Rework may be monitored by employee and tracked by cause (too much product, wrong product, foreign objects, etc.) to create notifications to management depending on severity and frequency of occurrence.” in Paragraph [0115], and “When implemented as a machine learning or artificial intelligence engine, the data analysis engine 285 may be trained using training data to, for example, to classify data or identify underlying relationships in the data.” in Paragraph [0050]), further comprising an audio device configured to provide audible alerts indicating start and stop of content capture to manufacturing packing operators (See “In some embodiments, the output devices 240 may also include alert systems 315. For example, in some embodiments, the controller 230 may raise alerts, alarms, warnings, or other notifications. For example, in some embodiments, the controller 230 may have detected a foreign object in meat. To ensure that the meat is not packaged or shipped, the controller 230 may raise an alert or alarm notifying suitable workers of the issue. In some embodiments, the alerts, alarms, warnings, or other notifications may be sent in a variety of ways. For example, in some embodiments, the alerts, alarms, warnings, etc. may be audible notifications and/or visual notifications. In some embodiments, the alerts, alarms, warnings, etc. may additionally or alternatively include automatically sending an email, text message, phone call, or other notifications to appropriate workers.” in Paragraph [0057]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include wherein the artificial intelligence model is specifically trained to detect packing anomalies using manufacturing packing process data and is configured to perform real-time analysis during active packing operations, further comprising an audio device configured to provide audible alerts indicating start and stop of content capture to manufacturing packing operators, as taught by Birkhofer, in order to improve quality and customer’s satisfaction.
Regarding Claim 11, SHAKES in view of Birkhofer teaches all the limitations of Claim 8 as described above. SHAKES also teaches wherein the support arm comprises an adjustable mount for positioning the content capture device (See “In some embodiments, the image capture devices 310 may be rigidly fixed in place, while in other embodiments, they may be configured to move or pan about in order to better capture images of the entire order processing.” in Paragraph [0053]).
Regarding Claim 13, SHAKES in view of Birkhofer teaches all the limitations of Claim 8 as described above. SHAKES does not explicitly teach; however, Birkhofer teaches wherein the artificial intelligence model is configured to detect at least one of: missing items, incorrect items, improper packing technique, or inadequate protective packaging (See “At operation 595, the controller 230 determines if there air or void inside the vacuum sealed package. In some embodiments, the controller 230, and particularly the data pre-processing engine 280 of the controller, may pre-process the image data. For example, in some embodiments, if multiple images (and/or videos) are captured for each piece of meat, the controller 230 may select an appropriate image (and/or video) for analysis (e.g., an image and/or video that shows a certain angle of the meat, shows a certain surface of the meat, etc.), combine multiple images (and/or videos) to obtain a suitable image and/or video (e.g., if none of the images are entirely suitable, the controller may stitch together multiple images to obtain a suitable image), modify images and/or videos (e.g., enhance the images to facilitate better analysis), convert video into static images, etc. In some embodiments, no pre-processing of the images (and/or videos) may be needed. Upon pre-processing (or if no pre-processing is used), the selected image data may be analyzed by the data analysis engine 285 of the controller 230. In some embodiments, the data analysis engine 285 may be previously trained to identify voids in the vacuum sealed package.” in Paragraph [0092], “From the mass spectrometer image, the controller 230 may identify any residual gas or oxygen in the vacuum sealed package. For example, in some embodiments, the controller 230 may determine an outer boundary of the packaging material of the vacuum sealed package and an outer boundary of the meat inside the packaging material. If the outer boundary of the packaging material and the outer boundary of the meat inside the packaging material do not match or have a gap that is greater that a pre-determined threshold, the controller 230 may determine that a void exists in the vacuum sealed package. In other embodiments, the image data may include other types of images that allow the controller 230 to determine the presence of air or gap in the vacuum sealed package.” in Paragraph [0093], “In some embodiments, products are monitored exiting packaging machines and where the product type can be identified/confirmed and the integrity of the package can be evaluated (looking for air in the bag). Products that do not pass this inspection may be automatically delivered to a rework station to be unpackaged and returned to packaging... Once product makes it to case packing (packaged product is placed in a box), each box is monitored for the correct products and the correct number/weight of products. If the bar code scanning of the label on the box doesn't match the products that are inside the case, then box is automatically rejected to a rework station to be inspected and corrected. Rework may be monitored by employee and tracked by cause (too much product, wrong product, foreign objects, etc.) to create notifications to management depending on severity and frequency of occurrence.” in Paragraph [0115], and “When implemented as a machine learning or artificial intelligence engine, the data analysis engine 285 may be trained using training data to, for example, to classify data or identify underlying relationships in the data.” in Paragraph [0050]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES to include wherein the artificial intelligence model is configured to detect at least one of: missing items, incorrect items, improper packing technique, or inadequate protective packaging, as taught by Birkhofer, in order to improve quality and customer’s satisfaction.
Regarding Claim 14, SHAKES in view of Birkhofer teaches all the limitations of Claim 8 as described above. SHAKES also teaches wherein the remote data store is a cloud-based server configured to provide secure access to the captured content and metadata for authorized users (See “Control system 300 may send the customer an electronic notification, such as an email message, instant message, cell phone picture, text message, or other electronic notification, including one or more captured images of the order being processed, and/or the electronic notification may include a link, URL, or other reference usable to access the captured images. Control system 300 and/or web server 600 may apply various security and/or authentication measures to prevent unauthorized access to captured images of orders being processed, esp. in embodiments where a customer's name and/or address may be visible in one or more of the images. The electronic notification may also include information regarding any such security measures and may also, in some embodiment, include all or partial authentication credentials necessary for accessing captured images. For example, in one embodiment, the electronic notification may inform the customer to login to web server 600 using a name and password the customer had previously setup, such as when originally placing the order. In another embodiment, however, login information, or other authentication credential information may be included in the electronic notification, or in separate electronic notification.” in Paragraph [0080] and “In some embodiments, captured visual verification data may be processed or enhanced before provided to a customer or other interested party. For instance, additional textual or graphic information may be added to captured images before the images are made available to a customer, either by the image capture device that captured the image or by control system 300, according to various embodiments. For example, in one embodiment, the date and/or time that an image was captured may be added to the captured image. Additionally, a customer name, or ID may be added to a captured image. Alternatively, information such as shipping personnel's names, packing station identifiers, the shipping weight of a package, environmental data, product identifiers, and/or order identifiers may also be added to the captured images. In general, any type of additional data or information that is relevant to and/or characteristic of an aspect of order processing may be added to or included with captured visual verification data. Such additional information and/or indicators may be included in, inserted into, overlaid over, or otherwise added to captured videos and still images in any of numerous ways, as is well understood in the art. The modification or enhancement of captured images may be performed manually, automatically, or as a combination of both. As various methods of image manipulation are well known in the art, and as any suitable image manipulation method may be used to include additional information in captured images and/or video, those methods will not be discussed herein. In some embodiments, however, captured or collected data may not be enhanced so that a customer or other party is provided with raw, unaltered data.” in Paragraph [0085]).
Regarding Claim 15, SHAKES in view of Birkhofer teaches all the limitations of Claim 8 as described above. SHAKES also teaches wherein the computing device is further configured to: generate the packing record comprising the captured content, metadata, any documentation associated with one or more packed cartons, and any detected anomalies; associate the packing record with shipping information for the carton; and make the packing record accessible to recipients of the carton (See “Control system 300 may receive the captured images and store them in a captured image database and associate them with the order being processed… In one embodiment, control system 300 may capture images of the finished, sealed, addressed order being loaded on delivery vehicle 360.” in Paragraph [0058], “After an order has been placed, the order may be processed at an order fulfillment center, as described above… Control system 300 may, according to one embodiment, receive the captured images, associate them with the order, and store them in image database 610 where web server 600 may make them available over the Internet 110. For example, in one embodiment, a separate web page may be automatically and dynamically generated for each order, while in another embodiment, such a web page may only be generated once the customer attempts to access the images, as will be discussed below.” in Paragraph [0079], “FIG. 7 illustrates an image database of captured images such as may be used to store images on web server 600, according to some embodiments. In other embodiments, however, such a database may store other types of captured data, such as audio, date/time, environmental data, or other data, either instead of or in addition to captured image data. In some embodiments, captured visual verification data may be processed or enhanced before provided to a customer or other interested party. For instance, additional textual or graphic information may be added to captured images before the images are made available to a customer, either by the image capture device that captured the image or by control system 300, according to various embodiments. For example, in one embodiment, the date and/or time that an image was captured may be added to the captured image. Additionally, a customer name, or ID may be added to a captured image. Alternatively, information such as shipping personnel's names, packing station identifiers, the shipping weight of a package, environmental data, product identifiers, and/or order identifiers may also be added to the captured images. In general, any type of additional data or information that is relevant to and/or characteristic of an aspect of order processing may be added to or included with captured visual verification data. Such additional information and/or indicators may be included in, inserted into, overlaid over, or otherwise added to captured videos and still images in any of numerous ways, as is well understood in the art.” in Paragraph [0085], and “As opposed to, or in addition to, capturing data for visual verification of order processing, as described above, data may be captured at one or more receiving areas in an order fulfillment center or materials handling facility. For instance, FIG. 10 illustrates capturing of images for a shipment of items arriving at a materials handling facility, according to some embodiments. Captured images of a container arriving at a order fulfillment center or materials handling facility may be reviewed to determine any defects in the container, such as damage to the container or defects related to an item or items in the container. For example, container 1000 may be delivered and one or more image capture devices 310 may capture images of the container being unloaded and unpacked, according to some embodiments. As mentioned above, captured images may include images of items being processed before an order has been placed for the item. For example, images may include one or more images of an item being received, inspected and stocked in inventory, according to one embodiment. Including captured images of items being received, inspected, and/or stocked may provide a customer with confidence that an item has been properly processed through the entire materials processing facility. In some embodiments, captured images of items being unpacked from a shipping container may be reviewed to determine any of various possible defects in the shipment. For example, reviewing captured images may help identify damaged, missing, and/or extra items in a shipment.” in Paragraph [0097]).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over SHAKES in view of Birkhofer and Eckman.
Regarding Claim 12, SHAKES in view of Birkhofer teaches all the limitations of Claim 8 as described above. SHAKES in view of Birkhofer does not explicitly teach; however, Eckman teaches wherein the computing device further comprises a lockdown application configured to restrict access to applications other than a packing monitoring application (See “Thus, for example, expansion memory 2474 may be provided as a security module for device 2450, and may be programmed with instructions that permit secure use of device 2450. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.” in Paragraph [0227] and Fig. 24).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of SHAKES in view of Birkhofer to include wherein the computing device further comprises a lockdown application configured to restrict access to applications other than a packing monitoring application, as taught by Eckman, in order to make the system more efficient and effective.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAYAR M KYU whose telephone number is (571)272-3419. The examiner can normally be reached Mon-Fri 9:00 am - 6:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jeffrey Zimmerman can be reached at 571-272-4602. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.M.K./Examiner, Art Unit 3628
/GEORGE CHEN/Primary Examiner, Art Unit 3628
Prosecution Insights