{"id":417,"date":"2013-06-05T01:09:22","date_gmt":"2013-06-05T07:09:22","guid":{"rendered":"http:\/\/www.onecle.com\/blog\/?p=417"},"modified":"2013-06-05T01:09:22","modified_gmt":"2013-06-05T07:09:22","slug":"sec-edgar-robots-txt","status":"publish","type":"post","link":"https:\/\/www.onecle.com\/blog\/2013\/06\/05\/sec-edgar-robots-txt\/","title":{"rendered":"SEC EDGAR Robots.txt"},"content":{"rendered":"

For a long time, a lot of data in securities filings was hidden by obscurity. Sure, the SEC offered a full text search<\/a> of EDGAR filings, but it only spanned the last four years. And, if you actually tried to use the full text search, it wasn’t too fruitful an experience.<\/p>\n

Here’s a search for offer letter<\/a>:<\/p>\n

\"SEC<\/p>\n

Not particularly illuminating.<\/p>\n

However, something changed about a year ago. On May 1, 2012, the robots.txt<\/a> file from the sec.gov website looked like this:<\/p>\n

User-agent: *
\nDisallow: \/Archives
\nDisallow: \/Archives\/bin
\nDisallow: \/Archives\/dev
\nDisallow: \/Archives\/etc
\nDisallow: \/Archives\/ftp
\nDisallow: \/Archives\/gopher
\nDisallow: \/Archives\/tmp
\nDisallow: \/Archives\/usr
\nDisallow: \/cgi-bin
\nDisallow: \/bin
\nDisallow: \/oursite\/previews
\nDisallow: \/edgar\/vprr<\/p>\n

Source: Internet Archive<\/em><\/p>\n

On the following day, the robots.txt<\/a> file looked like this:<\/p>\n

User-agent: *
\nAllow: \/Archives\/edgar\/data
\nAllow: \/Archives\/edgar\/vprr
\nDisallow: \/Archives\/bin
\nDisallow: \/Archives\/dev
\nDisallow: \/Archives\/etc
\nDisallow: \/Archives\/ftp
\nDisallow: \/Archives\/gopher
\nDisallow: \/Archives\/tmp
\nDisallow: \/Archives\/usr
\nDisallow: \/cgi-bin
\nDisallow: \/bin
\nDisallow: \/oursite\/previews
\nDisallow: \/Archives\/edgar\/vprr\/XXXX
\nDisallow: \/Archives\/edgar\/vprr\/vprr_removal
\nDisallow: \/Archives\/edgar\/vprr\/bin<\/p>\n

Source: Internet Archive<\/em><\/p>\n

The big change was this line that allowed search engines, such as Google, to index EDGAR data:
\nAllow: \/Archives\/edgar\/data<\/p>\n

Now you can use Google to search for an offer letter<\/a> in the EDGAR archives. The results are much better than the SEC’s full-text search.<\/p>\n","protected":false},"excerpt":{"rendered":"

For a long time, a lot of data in securities filings was hidden by obscurity. Sure, the SEC offered a full text search of EDGAR filings, but it only spanned the last four years. And, if you actually tried to use the full text search, it wasn’t too fruitful an experience. Here’s a search for […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-417","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack-related-posts":[],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/posts\/417","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/comments?post=417"}],"version-history":[{"count":0,"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/posts\/417\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/media?parent=417"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/categories?post=417"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.onecle.com\/blog\/wp-json\/wp\/v2\/tags?post=417"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}