With all due respect, this is incorrect information.
Regarding "delays":
The example used of seeing a product in stock then out of stock when added to the cart is due to caching. Websites fully render the code for a page, and refresh it every so often.
This saves the server from having to work to bundle up all the CSS, HTML, and JS each time a page is loaded, and instead, send a "pre-rendered" page. This is on the server side.
The browser does cache items, but properly designed and working websites break the cache when dynamic content is updated.
Additionally, browser caches are held by browsers that view a site multiple times. When browsing programmatically, simply using a get request, there is no client side caching.
"Scraping is irrelevant":
"Scraping" a website, is not a 15 year old technology, scraping just means programmatically collecting the website, and then using code to parse and break it apart into useful information.
It doesn't become irrelevant, its just a name for programmatically collecting data from a website.
"Scraping is illegal":
I don't care to discuss legal in depth on the internet, but there are many cases surrounding this.
From my analysis and my councils, PrimerPrimer is perfectly fine.
LinkedIn set some great precedent for scraping the web.
See:
https://parsers.me/appeal-from-the-...ourt-for-the-northern-district-of-california/
First let me again state that your business idea is awesome! Brilliant even! It truly is.
So logic would dictate that since it is an awesome idea with an attentive and waiting audience why hasn't someone else already done this? Why hasn't someone like AmmoSeek, a completely free service, invested in new technology platforms, built better database integration with partner companies, and done this already? Laziness? There are a ton of brilliant software developers out there. The technological know how has existed FOR YEARS!
Travelocity, and the hundreds of travel related clones since created, proved that technology can be quickly duplicated. So technology wasn't the reason hundreds of commercial travel sites didn't pop up overnight. There must have been a reason not related to unique software or technology in general. Well there was a huge stumbling and legal road block called American Airlines. American Airlines owned and provided ALL of the airlines with flight & reservation data through its proprietary Sabre reservation system. Every airline in the world had to license with Sabre to use this data.
(1996: Travelocity.com is launched as a joint venture of Sabre Interactive and Worldview Systems Corp.)
Delays
Actually my information is 100% correct.
So are your definitions provided as to how the Internet works for displaying web pages from the server side through to the browser side. No dispute from me at all. You've described exactly how server side and browser caching work and why they are used.
However,
I wasn't describing caching at all. I was describing patented, security based server content controls using proprietary software to deliver database controlled content for specific timed delivery. This patented technology has absolutely nothing to do with server side or browser caching. It does have everything to do with website security and server load. Specifically combating bots. Bots don't just crash a single category or website page. They dramatically affect the entire consumer experience and are costing retailers millions a year now. What I am describing happens before the code is delivered to the server for rendering which you have perfectly defined.
I am not knocking you or your ideas although I can certainly understand how my comments must appear to anyone wanting to protect their business and business idea. The patented controls I mention above were designed to combat the very real problem you wrote about and acknowledged in your own post below referring to scraping bots:
"This is actually a major problem. We usually send a text 1-2 minutes before sites start getting slow then crash. This is all from other bots too. It gotten pretty wild."
"Scraping is irrelevant":
Again you have defined website scraping perfectly: its just a name for programmatically collecting data from a website.
What makes it irrelevant is how you have described your use of it.
"Scraping is illegal":
As you have described your process and planned use of this data collection technique it is illegal. You have offered the entirely wrong court case (which is still pending by the way) for your business model. Why is this? The court case you present is in regards to whether a 3rd party can "scrape" the LinkedIn website to gather the PUBLIC PROFILES of LinkedIn members.
The key wording is PUBLIC PROFILES. LinkedIn like Facebook are platforms. Even Google is a platform. They did not create nor do they own their member's content. Exactly like this website. It's 100% created by members and members agree with each website's respective terms of service that ALL of their created content is available publicly to anyone. There is no expectation of privacy and no copyright protection as it pertains to public profile information use on each website. Each website can and does freely sell or share this member created profile information with advertisers and other platform users.
Midsouth Shooters Supply, Basspro, Midway USA, Cabelas, and Scheels are all content publishers and not PUBLIC entities. They are exactly like US News, Fox News, The Federalist, and millions more. Their published online content i.e. all text, charts, images, product pricing, and product availability,
is not in the public domain for commercial reuse. It is simply available for viewing by the public in general. Their ecommerce websites are simply extensions of their corporate identities and their entire website content is covered by US copyright law. Period. No one, and this includes your current business model, can independently "scrape" content from any online publisher's website, blog, etc.
for commercial reuse without the express written permission to do so. Regardless of content manipulation techniques used nor how re-displayed in the future
any 3rd party commercial use of copyrighted materials, including online published material, is illegal. Fair use does not apply per multiple court rulings.
You have a great idea! Done correctly I would sign up in a minute flat online. I am sure you will keep pursuing it. Perhaps creating an active publishing partnership with the same companies that you wish to currently "scrape" their data. With direct access to their live inventory databases you wouldn't need to "scrape" anything and it could become an instant,
ACCURATE, clearing house for a huge market that is inefficient for these companies to pursue on their own.
Costco doesn't make their money selling products in their stores. Their real profit is in selling memberships! Accurate, live inventory sells memberships.
I only took the time to comment originally due to my interest in reloading, shooting, and my lifetime support for entrepreneurs.
My opinions and comments are just that, opinions and comments. However, I have also spent many years in court working for ecommerce businesses and their lawyers.
I would suggest only that "scraping" websites using bots is not the way to build your business if you desire long term growth. I know this for a fact. Currently three major, exclusively online retailers, with the help of the DOJ Intellectual Property Division, have successfully filed lawsuits against two companies utilizing bots to crawl their ecommerce websites. The DOJ is utilizing technology which unequivocally proves who owned the bots. Advanced IP and tracing software that doesn't care how you try to filter the attack. They are seeking millions in monetary damages for lost revenue due to the website attacks (these laws have been on the books since 2001 when Clinton signed them into law.
Edit: Law initiated under Clinton but actually signed by Bush). And the DOJ is enforcing federal copyright law to further discourage future bot use. Copyright law alone will generate millions in fines.
FindLaw's article on Hacking Laws and Punishments. For more information on this and related topics, see FindLaw's section on Cyber Crimes.
www.findlaw.com
Bots have also been successfully categorized as denial of service attacks DoS
"In this form of hacking, an intruder floods the system or servers with traffic (bots are considered traffic), denying access to legitimate users. Florida
penalizes this more severely, categorizing this crime as a felony in the first degree."
My post began and remains not as an attempt to disparage or insult you or your idea. Just an opportunity to share some ideas. I thought what is written below was great which is the only reason I took so much of my time to comment, I also understand that the people on the forum are a great initial market test:
-----------------------------------------------------------------------------------------------
"How it started
A few months ago my buddies and I began laying the foundation for starting a commercial reloading operation. Quickly, we realized how bad the primer shortage had become.
After a bit of digging, many hours on hold, and chatting with a lot of different companies I realized something interesting.
While significantly less, the supply of primers to retails has not stopped, and many weeks 1 or 2 retailers will actually get primers in-stock online. (But you do have to be quick, they sell out in minutes)
So, we got to work. We built some code that will check about 23 different retailers at a very fast interval and text us a link when they are in-stock.
Over the last 30 days of using our system we were able to get enough primers to not worry about proceeding with our little commercial reloading operations."
-----------------------------------------------------------------------------------------
I will not respond again. Good luck to you and I wish you success!
