Crawling your store
You can use Web Bot Auth to securely authorize your own crawlers, scripts, or tools to access your public Shopify online store. Web Bot Auth relies on HTTP message signatures, which are generated in your Shopify admin. You must include the signature in the HTTP headers of each request so that the request can be verified as coming from an authorized source.
You can use signatures with automated first-party or third-party tools that access your online store for accessibility and SEO audits, automated testing, data analysis, and similar use cases. Your store can be indexed by search engines and large language models (LLMs) without signatures.
On this page
Signature features
Signatures have the following features:
- Secure authentication: You can generate cryptographically secure signatures for your crawlers.
- Expiration control: You can set signatures to expire after a specific period. The maximum period is 3 months.
- Centralized management: You can manage and view all your signatures in one place.
- Custom naming: You can give each signature a custom name to identify it.
- Domain scoping: Each signature is used for a specific domain that must be connected to your Shopify store.
Considerations for creating signatures
Before you create and use signatures, review the following considerations:
- Crawling your store with signatures is intended only for analyzing your online storefront. Signatures don't grant access to your Shopify Checkout due to the additional layers of checkout security.
- Use clear and descriptive names for your signatures.
- Signatures expire automatically and can't be renewed. Create a new signature and update your tools after a signature expires.
- Each signature is used for one domain, so you must specify which connected domain the signature will be used for. You can only select domains that have been connected to Shopify.
Create a signature
You can create and manage signatures directly in your Shopify admin.
From your Shopify admin, go to Online Store > Preferences.
In the Crawler access section, click Create signature.
In the Name field, enter a descriptive name for your signature.
In the Domain field, select the domain that you want to use this signature for.
In the Valid for section, select an expiration period for your signature.
Click Create.
Optional: To copy your newly created Signature-Input and Signature values, click Copy next to each value.
After a signature expires, it is no longer valid. You can't renew an expired signature, but you can create a new one.
Add a signature to requests
You can copy your signatures, and then add them to the HTTP headers of your requests. Each signature consists of 3 HTTP headers that you must include.
| Header name | Header value |
|---|---|
| Signature-Input | Add your signature input value. |
| Signature | Add your signature value. |
| Signature-Agent | The Signature-Agent value is a string containing a valid URI of the agent. Use the "https://shopify.com" (with quotes) for this value. |
For instructions on adding custom headers, refer to the following documentation:
View signatures
You can view the following information about your existing signatures in your Shopify admin:
- Signature name
- Associated domain
- First few characters of Signature-Input and Signature values
- Signature-Agent value
- Signature expiration date and status
From your Shopify admin, go to Online Store > Preferences.
In the Crawler access section, review information about your signatures.
Troubleshoot signatures
Select the signature issue and follow the troubleshooting steps.
Troubleshoot authentication issues
If your crawler is receiving rate-limiting errors, then your signature isn't valid. Verify the following information:
- Confirm that you're using the correct signature format in the HTTP headers.
- Ensure that the Signature-Input, Signature, and Signature-Agent headers are sent with every request.
- Verify that the signature hasn't expired. Review the expiration date in your Shopify admin.
- Crawl the domain that's associated with the signature.
Recover lost signatures
If you lost your signature, then you can recover it by completing the following steps:
- From your Shopify admin, go to Online Store > Preferences.
- In the Crawler access section, locate your signature in the list.
- Click Copy to copy the Signature-Input, Signature, and Signature-Agent values to your clipboard.
Change a signature expiration date
You can't change the expiration date of your signature after you create it. To set a different expiration date, create a new signature with the preferred expiration date, and then update your services to use the new signature.
The old signature is still valid and continues to work until the original expiration date.